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Executive Summary 


The project 


This report evaluates a developmental project designed by School 21 and the University of 
Cambridge to improve Year 7 students’ oracy skills. The project involved developing an Oracy Skills 
Framework, which sets out the physical, linguistic, cognitive, and social-emotional oracy skills 
required by students for education and life. Other components which were informed by this framework 
are: 


e adedicated Year 7 oracy curriculum comprising weekly oracy lessons; 
e racy in every lesson; 

e = building a whole school oracy culture; and 

e an Oracy Assessment Toolkit. 


These components were piloted and further developed with Year 7 students within School 21 from 
September 2013 to July 2014. During the final stages of the project the components were brought 
together to create an ‘Oracy Curriculum, Culture and Assessment Toolkit’ that can be adopted by 
other schools and a website was created, Voice 21 (http://voice21.org/), containing guidance and 
resources for schools using the Toolkit. 


This report focuses on an evaluation of: 


1. The approaches and materials which formed the Oracy Curriculum, Culture and Assessment 
Toolkit, including an indicative impact finding on the impact on Year 7 pupils in School 21. 

2. What further development of the Oracy Curriculum, Culture and Assessment Toolkit is 
needed and would enable a more robust evaluation of its impact. 


What did the project find? 


Oracy Skills Framework - The Oracy Skills Framework provided an appropriate and effective 
structure to support the design, review and refinement of School 21’s Year 7 oracy curriculum. It 
effectively underpinned the development of an assessment tool which supports diagnostic and 
formative assessment in oracy, and tracking of students’ progress. 


Year 7 oracy curriculum - The curriculum appears to have been well designed to provide sound 
foundations for the development of oracy skills with particular strengths in supporting persuasive talk 
and talk for presentational purposes as well as in formal contexts. A number of elements remain 
under development and a strengthening of the cognitive strand, including provision for exploratory 
talk, is required. The curriculum also needs to emphasise the need to address oracy within a diverse 
range of informal as well as formal contexts and explore the appropriateness of talk to context. 


Oracy in every lesson - The commitment to promote oracy across the curriculum, and in every 
lesson, has ensured that School 21 staff and students possess and utilise a shared language for 
oracy and are familiar with a range of approaches for organising, promoting and reviewing talk. 
Students have opportunities to use talk within a diverse range of motivating contexts and for different 
purposes. However, further consideration of how to engage students in exploratory talk and how this 
supports learning across the curriculum is required. 


Whole school oracy culture - The multi-stranded approach to embedding oracy across School 21 
has been effective in generating commitment from students and staff. 


Oracy Assessment Toolkit - The Oracy Assessment Toolkit provides a useful tool for measuring 
students’ oracy skills along with a range of support materials which if used appropriately, following 
training, should enhance the reliability of the tool. 
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Indicative impact - Findings from the pre and post Ravens Progressive Matrices tests, administered 
by the evaluators, showed no impact of the intervention on student's non-verbal reasoning skills. 
However, tests undertaken by the University of Cambridge using the prototype Oracy Assessment 
Toolkit developed for this project provided a tentative indication that the intervention had a positive 
impact on students’ oracy skills. As explained below neither of these tests were able to provide 
conclusive evidence of the impact of the intervention. 


How was the project conducted? 


The evaluation was undertaken alongside the development of the Oracy Curriculum, Culture and 
Assessment Toolkit and is therefore primarily formative, with the main purpose of supporting the 
further development of the project. These formative findings are informed by a combination of 
quantitative and qualitative data collection including: observations of oracy teaching, teaching of other 
subjects and whole school oracy activity; interviews and focus groups with the School 21 and 
University of Cambridge project teams, School 21 senior leaders, oracy teachers, teachers of other 
subjects, and students; and a review of the Voice 21 website and other documentary evidence. 


It was not appropriate at this initial stage of development to attempt to provide a robust measurement 
of impact. Two indicative quantitative measures of impact were captured: 


1. The Raven's Progressive Matrices Test - a non-verbal reasoning test. Pre and post test 
results at the beginning and end of the academic year for 20 Year 7 students from School 21 
were compared with 18 Year 7 students from a comparison school with similar characteristics. 

2. The prototype Oracy Assessment Toolkit created by the University of Cambridge. This was 
used with 12 Year 7 students in School 21 and 12 Year 7 students in a comparator school on 
a pre and post basis spanning five months within the academic year. 


A number of factors severely limit the confidence that can be placed in the measurement of impact 
that was possible during this project. These include: the lack of an existing standardised test that 
accurately measures the oracy skills that the project was setting out to develop; impact measurement 
being undertaken at the same time that the project was being developed; and the project only being 
implemented in one school. 


How much does it cost? 


This has been a developmental project in one school and therefore, at this stage, there is not an 
established cost for the intervention and an assessment of cost effectiveness cannot be made. School 
21 estimates that the cost of providing training for local schools would be approximately £6,000 per 
school. In addition, participating schools would need to fund participants’ travel and any necessary 
cover costs. 


Training activity 


One full day of CPD including visit to School 21 for all school staff 


Four twilight sessions for teachers delivering the dedicated oracy Year 7 £2,000 
curriculum 
Attendance at three focus groups by a senior leader and one nominated oracy 


lead teacher (assuming 6 schools participate in the focus groups) 
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The Voice 21 website resources and the assessment tool resources on the University of Cambridge 
website are freely available to any school. 


Key Conclusions 


1. 


The Oracy Skills Framework provides a useful tool for schools wishing to review and 
develop their approach to oracy. The associated Oracy Assessment Toolkit provides 
teachers with a tool that can be used diagnostically and to track students’ progress in 
developing oracy skills. 


The Oracy Curriculum, Culture and Assessment Toolkit as implemented in School 21 
appears to provide a sound foundation for the development of oracy skills, with particular 
strengths in supporting persuasive talk and talk for presentational purposes and in formal 
contexts. This multi-stranded approach may demand some fundamental shifts in approach 
for any new school adopting it, including allocating dedicated curriculum time, cultural 
changes and whole staff training. 


Further refinement of the curriculum and associated resources is required to highlight the 
role, nature and development of exploratory talk and to ensure diverse opportunities for 
oracy, formal and non-formal, are provided. The supporting website requires development 
to provide an effective professional development resource for other schools. 


It was not possible at this stage of development to provide a valid measurement of impact. 
Piloting in other schools and further research would be required prior to a randomised 
control trial of the intervention to establish a stronger evidence base on the impact on oracy 
skills and attainment across subjects. Research is also required on how the intervention is 
interpreted in other schools, and on the opportunities and barriers that arise when 
implementing these approaches at other sites. 
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1. Introduction 


1.1 Project 


This development project comprised two main phases: a design phase from January to July 2013 and 
a pilot stage which included further developmental work from September 2013 to July, 2014. 


In the first phase the University of Cambridge developed an Oracy Skills Framework (Appendix 1), 
which sets out the oracy skills needed by students for learning and life, and an Oracy Assessment 
Toolkit. The Oracy Skills Framework has informed the design of all elements of the project. During 
phase one School 21 began to codify the approaches that they were already taking to support 
students’ oracy and developed further strategies and resources to support the development of oracy. 


During the second phase School 21 piloted three interrelated approaches to developing oracy, 
namely: a dedicated Year 7 oracy curriculum where one lesson a week is spent teaching oracy; oracy 
in every lesson; and a whole school oracy culture. In addition the Oracy Assessment Toolkit 
developed in the design phase was tested by the University of Cambridge in School 21. Further 
development and refinement work on all aspects of the project continued throughout the pilot phase. 


During the final stages of the project School 21 brought together the project strands to create an 
‘Oracy Curriculum, Culture and Assessment Toolkit’ that can be adopted by other schools. This 
Toolkit comprises: 


e The Oracy Skills Framework — which sets out the physical, linguistic, cognitive and social 
and emotional oracy skills that students need to succeed in learning and life. 

e A dedicated Year 7 oracy curriculum — including units, lesson plans and resources for 
weekly oracy lessons. 

e Oracy in every lesson — including strategies for developing oracy in other subjects and 
enhancing learning in other subjects by using oracy approaches. 

e Awhole school oracy culture — strategies and techniques to embed an oracy culture. 

e The Oracy Assessment Toolkit — which can be used by teachers to track students’ progress 
against the skills in the Oracy Skills Framework. 


To provide guidance and resources for schools wishing to use the Oracy Curriculum, Culture and 
Assessment Toolkit School 21 developed a website Voice 21 (http://voice21.org/). This is linked to, 
and supplemented by, a _ University of Cambridge webpage  (http://www.educ.cam.ac.uk/ 
research/projects/oracytoolkit/) that provides additional resources and guidance on the Oracy Skills 
Framework and the Oracy Assessment Toolkit. 


1.2 Background evidence 


The intervention aims to establish a whole school approach to developing oracy. Research over many 
years has demonstrated the central role of talk in learning across the curriculum (Barnes and Todd, 
1976; Mercer, 2008; Alexander, 2001) and the need to support students in developing a broad 
‘repertoire’ of talk. An oracy curriculum that supports talk for learning might therefore be expected to 
include a range of opportunities for different kinds of talk. As Alexander (2012: 4) argues, 


Students need, for both learning and life, not only to be able to provide relevant and focused answers 
but also to learn how to pose their own questions and how to use talk to narrate, explain, speculate, 
imagine, hypothesise, explore, evaluate, discuss, argue, reason and justify. 


In a presentation given during a Department for Education seminar on oracy, for example, Alexander 
(2012) argued that there is ‘robust evidence’ for the relationship between the quality of classroom talk 
and attainment in core subjects. Indeed, a series of quasi-experimental studies have correlated: 
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e group discussion with learning, with impacts for example on scientific understanding (e.g. 
Rivard and Straw, 2000) 

e mathematics and reading (Kutnick and Berdondini, 2009) 

e anexplicit focus on exploratory talk with reasoning (Mercer et al., 1999). 


Employers have also called for the need to ensure that students develop effective communication 
skills, and the ability to talk for persuasive purposes has been linked to effective participation in civic 
and social life (Hartshome, 2011). 


Many teachers and schools however remain uncertain about the explicit teaching and assessment of 
oracy. Understanding progression and attainment in oracy is problematic as talk is by nature 
ephemeral and consequently difficult to record and analyse, and people will speak differently in 
different contexts. Consequently the efficacy of talk needs to be assessed in relation to purpose, 
audience and context. There have been various moves to raise the profile and status of oracy across 
the curriculum in England including The National Oracy Project (Norman, 1992) and the inclusion of 
speaking and listening guidelines for Assessing Student Progress (QCDA, 2008). However, as the 
University of Cambridge team highlights (Mercer et al., 2014), oracy has received far less attention 
than reading and writing and its scope has been reduced in the new National Curriculum (DfE, 2014). 
This is particularly the case at Key Stage 3, where the programme of study foregrounds opportunities 
for formal speaking, such as presentations and debates. The new National Curriculum includes less 
emphasis on talk for learning than the previous curriculum, although the role of talk across the 
curriculum is acknowledged, particularly in Key Stages 1 and 2. 


With these developments in mind, it is worth noting that this report sometimes distinguishes between 
the development of talk for presentational purposes and talk for learning and reflection. We recognise 
that this distinction is rather problematic. The very process of deciding how best to communicate 
ideas, for example, is likely to involve clarification and reflection on learning that itself may enable 
students to develop greater understanding. However, we suggest this distinction is helpful as it 
acknowledges different emphases that we might expect from an oracy curriculum and this has 
implications for what we expect from students’ talk in different contexts. Barnes and Todd, for 
example, in their seminal research into classroom group work articulate this distinction by 
differentiating between the exploratory talk associated with learning which ‘is usually marked by 
frequent hesitations, rephrasings, false starts and changes of direction’ (1976: 28) and the ‘final draft 
language [in speech or writing] is the contrary of exploratory: far from accompanying (and displaying) 
the detours and dead-ends of thinking, it seeks to exclude them and present a finished article, well- 
shaped and polished’ (Barnes and Todd, 1976, p. 108). While ‘final draft language’ may be easier to 
evaluate (not least because it is easier to ‘hear’), repeated studies have suggested that the effective 
promotion of exploratory talk can have a significant impact on learning. An oracy curriculum therefore 
might be expected to make a significant contribution to promoting a ‘dialogic space’ (Wegerif, 2007) in 
which ‘different ideas, perspectives and understandings can be collectively explored, and material can 
be modified to record the development of a discussion and capture emerging ideas’ (Mercer and 
Littleton, 2007: 90). Influential work by Mercer and colleagues has generated guidelines and 
resources to support exploratory talk and a series of experimental studies have demonstrated the 
impact of this work (see Mercer and Littleton, 2007). Research involving multimodal analysis of 
collaborative learning has also highlighted how individuals draw on different modes such as gesture, 
posture and gaze as they negotiate meaning (Taylor, 2013). 


While further research is needed in this area (Howe and Abedin, 2013), the discussion above 
suggests there is a strong mandate for more effectively promoting oracy in schools and that an oracy 
curriculum might reasonably be expected to: 


e support the development of talk for learning; 
e provide opportunities for students to use talk effectively in diverse contexts and for a range of 
purposes and audiences including those associated with different subjects; 
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e equip students to reflect on their use of talk and make choices about the kind of talk that is 
appropriate to different contexts. 


The main purpose of this evaluation is to consider the potential of the Oracy Curriculum, Culture and 
Assessment Toolkit developed by School 21 and the University of Cambridge to make a valuable 
contribution to practice in each of these areas. 


1.3 Evaluation objectives* 
Research aims 
This is a formative evaluation which aimed to support the development of the project by providing: 


e indicative evidence of the potential of the Oracy Curriculum, Culture and Assessment Toolkit 
to impact on student attainment; 
e an independent view on the quality and fitness for purpose of: 
othe Oracy Skills Framework, the dedicated Year 7 oracy curriculum, approaches to oracy 
in every lesson and approaches to building a whole school oracy culture; 
o the Oracy Assessment Toolkit, methods of administering the tool and associated training 
materials; 
o the CPD and resources (on the Voice 21 website) provided to support the implementation 
of the intervention; 
O guidance on developing intervention approaches, materials and training that are 
replicable and potentially testable through a randomised control trial. 


Evaluation questions 


1. Impact 
1.1. What has been the impact of the pilot on the development of students’ oracy skills? 


2. Oracy Skills Framework 
2.1. How well does the Oracy Skills Framework represent the skills required by children and 
young people to succeed in education, employment and life? 


3. Dedicated Year 7 oracy curriculum, oracy in every lesson and whole school oracy culture 

3.1. To what extent do these approaches provide the resources needed to enable children and 
young people to develop the skills identified in the Oracy Skills Framework? 

3.2. What are the strengths and weaknesses of these approaches and the proposals for 
deployment, with a particular focus on their usability and impact on students’ engagement 
and learning? 

3.3. In what ways can these approaches and proposals for deployment be improved? 


4. Oracy Assessment Toolkit 
4.1. Is the assessment being administered consistently? 
4.2. How well does the assessment guidance support consistent administration of the test? 
4.3. How reliable and valid is the assessment tool as a measure of oracy skills? 
4.4. What changes need to be made to the assessment design, administration and guidance to 
ensure reliability and validity and consistent administration? 


5. Supporting CPD and resources 
5.1. To what extent do the design and delivery of CPD and the presentation of resources on the 
Voice 21 website fit with the current knowledge base on effective CPD? 
5.2. How effective is the CPD in developing: teachers’ knowledge and understanding of how to 
deliver the oracy curriculum/ implement oracy in the wider curriculum; the motivation and 


* Due to the developmental nature of the project, the initial intentions for the intervention design have evolved and changed 
over the project period. Consequently, the original evaluation aims and questions have been modified to ensure that the key 
components of the intervention, as they emerged, have been subject to independent evaluation. 
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confidence necessary to implement the curricula; and their ability to effectively implement the 
curricula? 
5.3. In what ways can the CPD delivery and the Voice 21 website be improved? 


6. Replicability of the intervention 
6.1. Are the outputs of the pilot usable as a set of inputs for use in other schools? Where are the 
risks to consistency and coherence, and how can they be overcome? 


1.4 Project team 
Delivery team: School 21 and the University of Cambridge 


The project delivery team comprises staff from School 21 and the University of Cambridge. School 21 
focused on the development, implementation and codification of: the dedicated Year 7 oracy 
curriculum; oracy in every lesson and the whole school oracy culture. In addition School 21 created 
the Voice 21 website to host the resources produced in this project and proposed a CPD package to 
support other schools wishing to develop their approaches to oracy. Delivery of the project at School 
21 was directed by Edward Fidoe (Chief Executive Officer - 21 Trust) supported by project manager 
Ben Mortimer. Daniel Shindler, a School 21 drama and oracy teacher, has led a core team of 
teachers in developing the oracy curriculum and resources. Holly Page was employed to populate the 
Voice 21 website with the materials produced during the project. The School 21 head and deputy 
head teachers have championed oracy across the school. The school has been supported by an 
expert advisory board. 


The University of Cambridge team comprises Professor Neil Mercer, Paul Warwick and Dr Ayesha 
Ahmed. The team developed the Oracy Skills Framework in consultation with a range of academic 
and professional experts. This included undertaking a review of research and innovation in oracy, 
including studies from a range of sub-fields including speech and language development and the 
teaching of English to speakers of other languages to underpin the development of the framework 
(see Mercer et al., 2014). In addition, the University of Cambridge team developed the Oracy 
Assessment Toolkit and tested this with students from School 21 and a comparator school. The team 
undertook a review of the validity and reliability of the tool, which is also reported in Mercer et al., 
2014. 


Evaluation team: Sheffield Hallam University 


The Centre for Education and Inclusion Research at Sheffield Hallam University was responsible for 
the evaluation design, qualitative data collection, administering of the Raven's Progressive Matrices 
Test, qualitative fieldwork, professional review of documentary evidence, analysis and reporting. The 
team comprised: 


Dr Bronwen Maxwell Project Director and CPD expert 
Professor Cathy Burnett Literacy and curriculum expert 

Dr John Reidy Test design and RCT expert 

Ben Willis Project Manager 

Sean Demack Statistical advisor on potential RCT design 


The Language and Literacy research group at Sheffield Hallam University contributed to the review of 
the Voice 21 website. 


1.5 Ethical review 


The evaluation was approved by the ethics committee at Sheffield Hallam University. Opt-in consent 
was sought from parents/carers prior to the trial at School 21. The rationale for seeking opt-in parental 
consent for the intervention group was that consent covered not only participation in the Raven's 
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Progressive Matrices tests but also lesson observations and student focus groups conducted by the 
evaluation team. In addition opt-out consent was sought directly from students. Opt-in consent was 
also gained from all staff who were interviewed or attended focus groups at School 21. At the control 
school it was not deemed necessary to request opt-in parental consent because student involvement 
related solely to testing. Opt-in consent was obtained from the head teacher of the control school and 
opt-out consent was obtained from parents. Students had the opportunity to opt out prior to the tests. 
All consent documentation was accompanied with a bespoke information sheet (Appendices 3-8). 
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2. Methodology 


2.1 Intervention 


The end point of this development project was the codification of an Oracy Curriculum, Culture and 
Assessment Toolkit and the production of the Voice 21 website to host the intervention resources. In 
this section we describe this intervention in the form that it is envisaged by School 21 in their final 
report (Fidoe, 2014). It is important to note that the evaluation was taking place while the Oracy 
Curriculum, Culture and Assessment Toolkit was being developed. 


The key components of the Oracy Curriculum, Culture and Assessment Toolkit are: 


e an Oracy Skills Framework 

e adedicated Year 7 curriculum 
e oracy in every lesson 

e  =oracy culture 

e the Oracy Assessment Toolkit. 


We describe each of these components in more detail below. 
The Oracy Skills Framework 


The Oracy Skills Framework (Appendix 1) identifies the skills needed to be able to communicate 
effectively in a range of settings and styles, using a wide vocabulary with fluency. The Framework 
isolates key components of spoken language, breaking them into four areas: physical, linguistic, 
cognitive and social and emotional. The Cambridge team identify these strands as ranging from 


...the ‘physical’ (voice projection, gesture and so on), ‘linguistic’ (using appropriate 
vocabulary, choosing the right register and language variety for the occasion), ‘cognitive’ 
(such as organising content well, taking account of the level of understanding of an audience) 
and ‘social and emotional’ (such as managing group activity, taking an active role in 
collaborative problem solving, and so on). (Mercer et al., 2014: 3) 


The Oracy Skills Framework is intended to be relevant in any context, formal or informal, within and 
beyond school, and underpins the design of the dedicated Year 7 curriculum, approaches to oracy in 
every lesson and the Oracy Assessment Toolkit. Explanation of the specific items in the oracy skills 
framework can be found in (Mercer et al., 2014: 53-57). 


Dedicated Year 7 curriculum 


One lesson a week is dedicated to teaching oracy throughout Year 7. The aim of this dedicated oracy 
curriculum is to provide students with a variety of contexts in which to develop all four skill areas. The 
curriculum continues to be constantly reviewed and developed and the supporting Voice 21 website 
continues to be updated. Outline plans, supporting resources and exemplar video clips have been 
developed for the four units of work that make up the dedicated oracy curriculum and are available on 
the Voice 21 website. These are: 


Unit 1: Finding our voice: This module aims to equip students with a range of strategies and 
protocols to support them in their listening, talking and ability to work effectively in group 
situations. It also familiarises students with the four strands of the Oracy Skills Framework. 


Unit 2: Performance poetry: Performance poetry is used as a vehicle to explore effective 
talking and listening in small groups. The module focuses particularly on the physical and 
social and emotional strands of oracy. 
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Unit 3: Persuasive techniques: In this module students explore a range of different formal 
and informal talk scenarios in which persuasive techniques are deployed and reflect on how 
they might best share their views with others. There is a focus on well-being and on the 
linguistic strand of oracy. 


Unit 4: Ignite: This module prepares students for a five-minute individual talk on a subject 
chosen by the student which is delivered without notes to an audience. Students are 
encouraged to use the techniques that they have acquired over the year and draw on skills 
from all four strands of the Oracy Skills Framework to deliver their speech. 


Appendix 2 outlines some of the ways in which School 21 has embedded the Oracy Skills Framework 
within the dedicated oracy curriculum. 


Oracy in every lesson 


School 21 has identified a number of teaching and learning strategies that can be used in all subject 
areas to support the further development of students’ oracy skills as well as to support learning in the 
subject. The intention is to use oracy approaches in every lesson. These include the techniques 
included in Unit 1 of the dedicated oracy curriculum, group work protocols, use of peer critique, ghost 
reading, collective writing and robust vocabulary teaching/use of tiered vocabulary. 


Oracy culture 


A key feature of the School 21 approach to oracy is the development of a whole school oracy culture. 
The Oracy Curriculum, Culture and Assessment Toolkit includes four initiatives to support a whole 
school oracy culture: 


Assemblies: Students sit in the round and engage in interactive activities, often in coaching 
groups of six or twelve students. These activities cover many of the oracy strategies taught in 
the dedicated Year 7 oracy lessons, for example types of talk, listening strategies and working 
in groups. Assemblies are also intended to be a CPD tool for teachers who are able to 
observe other teachers modelling oracy teaching. 


Coaching groups: Each year group is split into coaching groups formed of twelve students 
and one teacher who meet together on a daily basis. Students are encouraged to reflect on 
their learning and behaviour and coaching groups may be used to explore morals and values 
and discuss social issues. Oracy skills are developed through talking points and open 
questions that stimulate discussion and debate. 


E-portfolios: Instead of a traditional parents’ evening, students are required to draw on their 
oracy skills to deliver a selection of their best work to a panel comprising their parents, coach 
and a school governor and respond to the panel's questions. 


Story of learning: School 21 believes that the experiences that a student has at school 
should form a 'story of learning’. In addition to recounting this in their e-portfolio, students are 
required to speak passionately about their successes by recounting their story of learning 
when guiding visitors around the school and meeting with potential future students and their 
parents at open evenings. 


Oracy Assessment Toolkit 


The Oracy Assessment Toolkit is based on measuring the skills set out in the Oracy Skills 
Framework and comprises: 


e aset of three initial and end of year oracy assessment tasks 
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e asetof six additional tasks to be used for assessment for learning 
e arating scheme for assessing students’ performance on these tasks and for giving students 
feedback. 


Full details of the assessment tasks including instructions for teachers and for students, the rating 
scheme and rating proformas for teacher assessment and proformas for self and peer assessment 
(assessment for learning tasks only) can be found in Mercer et al. (2014) and at 
http://www.educ.cam.ac.uk/research/projects/oracytoolkit/. 


In summary the Oracy Assessment Toolkit includes three assessment tasks that can be conducted at 
the beginning and end of Year 7: a formal presentational speech; an instructional activity whereby one 
student enables another to complete a specific task; and a group discussion in which three students 
are asked to reach joint conclusions about a specific topic. The rating scheme enables teachers to 
give students a rating for each of the skills from the Oracy Skills Framework that are relevant to the 
task. Teachers can use this information to create a skills profile for each student and plan teaching to 
meet students' needs. Using this group of assessment tasks at the beginning and end of the year 
enables teachers to track students’ progress. The rating schemes as used at School 21 during this 
evaluation are included in Appendix 9. Finalised versions of the instructions and ratings sheets for all 
three tasks are available electronically on the Voice 21 website (http://voice21.org/) and 
http://www.educ.cam.ac.uk/research/projects/oracytoolkit/. Exemplar videos of assessment of the 
tasks that enable teachers to moderate their grading are available on the University of Cambridge 
website (http://sms.cam.ac.uk/collection/1830303). 


In addition to the three assessment tasks described above, the University of Cambridge team have 
developed six additional assessment for learning tasks which are designed to be used flexibly and 
adapted to fit with a school’s existing curriculum. Details of these and the associated rating schemes 
and proformas can also be found in Mercer’ et al. (2014) and at 
http://www.educ.cam.ac.uk/research/projects/oracytoolkit/. 


Supporting CPD and resources 


School 21 have proposed a package of CPD support for schools wishing to take on the full 
intervention. This comprises: 


e aone-day training session for all staff that includes a visit to School 21 and focuses on whole 
school oracy strategies and techniques and unit 1 of dedicated year 7 oracy curriculum 

e four twilight training sessions for lead oracy teachers focusing on Units 1-4 of the dedicated 
year 7 oracy curriculum 

e three focus groups staged over the first year of implementation. Each group would include 
one senior leader and one lead oracy teacher from approximately six different schools 
together with a senior leader and a member of the oracy team from School 21. 


In the longer term it is envisaged that schools who are trained by School 21 will go on to lead training 
for other schools. 


This CPD support is supplemented by websites that are freely available to anyone interested in 
developing oracy. The Oracy Curriculum, Culture and Assessment Toolkit can be found on the Voice 
21 website at http://voice21.org/ and additional information on the Oracy Assessment Toolkit can be 
found in the University of Cambridge project report (Mercer et al., 2014). Resources for undertaking 
the assessments and exemplar videos of assessment grading are available at 
http://www.educ.cam.ac.uk/research/projects/oracytoolkit/. 
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2.2 Evaluation design and methods 
Overall design 


As the research aims and questions outlined in Section 1.3 indicate, this was a formative assessment 
designed primarily to support the development of the Oracy Curriculum, Culture and Assessment 
Toolkit. 


The evaluation was undertaken in two stages. The first phase from February to August 2013 was an 
independent professional review of the plans and materials for developing and assessing oracy that 
were developed by School 21 and the University of Cambridge in preparation for using with Year 7 
students in School 21 in phase 2. In addition to examining documentary evidence the evaluation team 
conducted telephone interviews with the Project Director and Project Manager at School 21 and the 
University of Cambridge team. A further meeting was held with the University of Cambridge team to 
discuss their plans for developing the Oracy Assessment Toolkit and testing its reliability and validity. 
The findings of this review were reported in an interim report (Maxwell et al., 2013) and shared with 
the delivery partners and EEF. The report provided a set of questions designed to support 
development in the second phase of the project., In addition two overarching recommendations were 
made, namely: 


e toset out a clear framework of aims, objectives, outcomes and indicative content for the oracy 
curriculum that School 21 want to promote 

e to take stronger account of the need for replicability in the design of curriculum and CPD 
approaches and materials. 


In this report we focus on the second phase of the evaluation. This ran from September 2013 to July 
2014. In this second phase a dedicated Year 7 curriculum was delivered via weekly oracy lessons 
and a range of strategies were implemented to incorporate oracy in every lesson and underpin a 
whole school oracy culture. In addition, the Oracy Assessment Toolkit tasks were piloted. Throughout 
the year School 21 and the University of Cambridge team were engaged in reviewing and further 
developing the intervention. The data collection activities undertaken by the evaluation team in phase 
2 of the evaluation and timings are summarised in Table 2.1. 


Table 2.1: Data collection activities conducted by the evaluation team in phase 2 of the 
evaluation 
Date Data collection activity 


Sept 2013 Raven's Progressive Matrices pre-test administered in School 21 and one control 
school 
Nov 2013 Field visit to School 21: 


three observations: assembly; drama lesson; mathematics lesson 
three interviews: project lead, project manager, and English teacher 
two focus groups: two oracy leads; three teachers of other subjects 

Jan 2014 Meeting with University of Cambridge team to review the initial assessments made 
using the Oracy Assessment Toolkit and discuss their approach to measuring the 
validity and reliability of the assessment tasks 

Feb 2014 Field visit to School 21: 
one observation: oracy lesson 
four interviews: project leader and project manager; head teacher; oracy lead; oracy 
teacher 
two focus groups: seven pupils; eight teachers - oracy and other subjects 


June 2014 Field visit to School 21: 
two observations: oracy lesson; humanities lesson 


three interviews: project leader; oracy lead; teacher of another subject 
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two focus groups: seven pupils; three teachers - oracy and other subjects 


End June Raven's Progressive Matrices post-test administered in School 21 and control school 
2014 
July 2014 Assessment of videos of School 21 students undertaking the Oracy Assessment 


tasks by four secondary English teachers from outside School 21. This activity, 
managed by the evaluation team, combined an independent check of inter-rater 
reliability and a focus group to gather teachers’ views on using the Oracy 
Assessment Toolkit. 


July! Aug Telephone interviews: Project director, School 21 oracy teacher and teacher 
2014 responsible for developing the Voice 21 website 


Oct/Nov 2014 Review of: 


the Voice 21 website and the University of Cambridge's Oracy Assessment 
Toolkit website 


the final report produced by School 21 (Fidoe, 2014) 


the final report produced by the University of Cambridge team (Mercer et al., 
2014) 


Nov 2014 Telephone interviews: Project director and University of Cambridge team 


Measurement of impact 


As previously explained, the main purpose of the evaluation was to support the development of the 
intervention to a stage of being ready for use by other schools. It was not feasible at this initial stage 
of development, where the intervention was subject to continuous development and only delivered in 
one school, to provide a robust measurement of impact. However, indicative measures of impact were 
captured using the Raven's Progressive Matrices non-verbal reasoning test and the Oracy 
Assessment Toolkit created by the University of Cambridge. We discuss these and their limitations 
below. 


Raven's Progressive Matrices test 


To establish the impact of the intervention generally on learning beyond language-related attainment 
measures it was decided to utilise the Raven's Progressive Matrices as this is a widely used 
standardised non-verbal reasoning measure. It is also a measure that members of the University of 
Cambridge team have used in previous research to demonstrate the general impact on learning of an 
explicit focus on exploratory talk on attainment (e.g. Mercer and Littleton, 2007). 


To assess the impact of the intervention on School 21's students’ progress in non-verbal reasoning a 
control school was recruited from the same geographical location with similar characteristics. Initial 
testing using the Raven's Progressive Matrices took place in mid-September 2013 in both schools. 
This was the earliest point in the new academic year that the students were able to be tested and was 
early enough for there to be little impact of the oracy curriculum on non-verbal reasoning at School 
21. The students were then tested again using the Progressive Matrices in June 2014. 


Students in both School 21 and the control school were tested in September 2013. School 21 asked 
for consent from parents/carers of all 75 Year 7 students. However, despite repeated requests, 
consent forms were only returned from the parents/carers of 36 students (14 females, 21 males and 1 
student did not record gender). At the control school 19 Year 7 pupils were available (12 females, 7 
males) to complete the Raven's Progressive Matrices test on the scheduled day of the test. 
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The post test was conducted in both schools at the end of June 2014. At School 21, of the 36 
students who took the post-test 20 students (10 males, 9 females and 1 student did not record 
gender) were available on the day of the post-test. In the control school 18 (11 females and 7 males) 
of the original 19 students who were tested in September 2013 took part in the test in June 2014 
(Figure 2.1). 


Figure 2.1: Number of students given the Raven's Matrices at the pre-intervention stage and 
those who were then tested again post-intervention 


School 21 


e 36 students at initial testing 
e 20 students at follow-up 


Control School 


e 19 students at initial testing 
e 18 students at follow-up 


Students were tested on the Raven's Progressive Matrices on both occasions in groups with all 
sessions being introduced and overseen by the same researcher from the evaluation team. Testing 
sessions took between 45 and 60 minutes to complete. Only those students who completed the 
Raven's Matrices at pre- and post-intervention were included in the inferential statistical analyses. The 
answers were scored according to the standard scoring template and converted to standardised 
scores utilising the tables in the Raven's Progressive Matrices Manual. The standardised scores 
provide age-appropriate scoring norms to aid interpretation of student performance on this particular 
measure. 


A number of factors limit the confidence that can be placed in this measurement of impact. Although 
the test has been used as a measure of non-verbal reasoning skills it does not constitute a global 
measure of impact upon non-verbal skills (e.g. mathematical reasoning or spatial ability) but a specific 
measure of non-verbal reasoning. It also does not measure the full skill set covered by the Oracy 
Skills Framework that underpins this intervention. Indeed this is the rationale for the University of 
Cambridge developing an Oracy Assessment Toolkit as part of this project. The continuing 
development of the interventions between the pre- and post-test also reduces the confidence that can 
be placed in the findings. It is interesting to note that some students from both the intervention and 
control schools reacted critically to undertaking the Raven's test and genuinely questioned how it was 
relevant to talking or oracy. 


University of Cambridge pilot Oracy Assessment Toolkit 


The University of Cambridge used their pilot Oracy Assessment Toolkit to undertake an _ initial 
assessment of 12 students in School 21 and 12 students in a comparison school (a different school to 
that used as the control school for the Raven's testing) in October 2013 and a follow-up test in April 
2014. For full details of the assessment tasks, rating criteria and assessment recording proforma see 
http://www.educ.cam.ac.uk/research/projects/oracytoolkit/. Students undertook three tasks at the pre 
and post assessments. The tasks were: an individual presentation task, a group talking points task, 
and a paired instructional task. 


In this report we present the evidence from the University of Cambridge's analyses of these 
assessment results to provide a further indication of the impact of the Oracy Curriculum, Culture and 
Assessment Toolkit. Each assessment was graded as bronze, silver or gold. These ratings were 
converted to numbers as shown in Appendix 10. 
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The mean overall task ratings for each task for students from School 21 were then compared with the 
mean overall task ratings for the comparison school at pre and post test and the difference in means 
calculated. Further details of the analysis can be found in Mercer et al. (2014: 23-26). 


The Cambridge assessment tool has stronger validity as an impact of the oracy intervention than the 
Raven's Matrices test as it is designed to measure the skills in the Oracy Skills Framework. However, 
the Cambridge measurement also has limitations: the assessment tool itself was under development 
between the pre and post test and the sample sizes are small. In addition the assessment tool is not 
as yet a standardised assessment test. 


Qualitative data 


As Table 2.1 indicates, qualitative data were collected over the course of the pilot. This comprised 
three field visits which included observations of teaching and interviews and focus groups with oracy 
teachers, teachers of other subjects, students, school leaders and the project leaders. These were 
supplemented by telephone interviews with the delivery team at School 21 and the University of 
Cambridge team. In addition the evaluation team met with the University of Cambridge team to review 
the use of the Oracy Assessment Toolkit and the work being undertaken by the team to assess the 
validity and reliability of the assessment tasks. The evaluation team arranged for a group of 
secondary English teachers to use the Oracy Assessment Toolkit to assess videos of School 21 
students undertaking the assessment tasks to provide an independent measure of inter-rater reliability 
and to provide additional views on the Oracy Assessment Toolkit. The Voice 21 website, the 
University of Cambridge Oracy Assessment Toolkit website and associated materials, and the School 
21 and University of Cambridge final reports were reviewed by the evaluation team. The Language 
and Literacy in Education Research Group at Sheffield Hallam University also undertook a review of 
the Voice 21 website. 


The evaluation questions were used as a framework to organise the qualitative data and a thematic 
analysis of the data was undertaken in relation to each evaluation question. 


Participation and recruitment 


All 75 students in Year 7 in School 21 were engaged in the oracy intervention throughout the 
academic year 2013/14. The oracy intervention was an integral part of the overall curriculum and 
school experience provided for Year 7. Details of the numbers of students in School 21 and the 
control school who undertook the Raven's Matrices test and of the students whose progress in oracy 
was measured using the University of Cambridge pilot Oracy Assessment Toolkit are given above. 


Protocol violations 


At the time the protocol was produced the intended intervention was at a very early stage of 
development. It was therefore necessary to make some changes to the evaluation questions and data 
collection activities published in the protocol to ensure that the evaluation matched the emerging 
intervention design and the intervention activity that took place within the project period. The main 
changes are described below: 


e The protocol assumed that a bank of oracy resources would be developed in the design 
phase of the project and then piloted in School 21. To better match the developmental nature 
of the project and School 21's developing codification of the key approaches that support the 
development and use of oracy skills we have amended the evaluation questions that 
previously related to ‘a bank of interventions’ to focus on the dedicated Year 7 curriculum, 
oracy in every lesson and a whole school oracy culture. 

e The evaluation questions were also written on the assumption that a codified package of CPD 
activity would be delivered to teachers in School 21 as part of the pilot. While staff did 
participate in a range of professional development activities related to oracy during the pilot 
period this was an integral part of school CPD activity, rather than a discrete identifiable 
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activity. This limited the planned evaluation of the intended CPD package. Although 
observations of CPD activity were not undertaken, teachers were asked about what had 
supported their professional learning in relation to oracy teaching and learning in interviews 
and focus groups. 
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3. Findings: Impact 


In this section we present evidence on student progress from the pre and post Raven's Progressive 
Matrices tests and the pre and post University of Cambridge pilot Oracy Assessment tests. We 
address the following research question: 


What has been the impact of the pilot on the development of students’ oracy skills? 


As discussed in Section 2.2 we are only able to place limited confidence in the measures of impact 
presented in this section due to the developmental nature of the project, the location of the project in 
only one school, and the lack of a standardised test to measure the range of oracy skills that the 
project was seeking to develop. 


We also present qualitative evidence on the impact on student engagement and the practices that 
appear to underpin this engagement. 


3.1 Impact on students’ oracy skills and learning 


The means and standard deviations for the raw scores and age standardised scores for School 21 
and the control schools at the pre- and post-intervention stages for the Raven's Progressive Matrices 
are presented in Table 3.1. It is apparent from the table that the control school students appear to 
have performed better at the pre-intervention stage than the intervention school students. However, 
the difference between the schools was not statistically significant for either the raw scores (t(36) = 
1.05, p = .302, d = 0.35) or the age standardised scores (t(36) = 0.81, p = .423, d = 0.27). It would 
appear therefore that at the time of the initial testing there was no difference in non-verbal reasoning 
ability between the students in School 21 and those in the control school. 


The means and standard deviations for the post test as presented in Table 3.1 indicate that again the 
control school students performed better overall than the intervention school. Both schools improved 
in terms of raw scores from pre- to post-intervention assessment; however, the age standardised 
scores for both schools reduced a little over this time period. The standard scores are important as 
these are age appropriate scores and thus the post-intervention scores reflect the fact that the 
students were older at this time of testing. For us to demonstrate the impact of the dedicated Year 7 
curriculum, oracy in every lesson and the whole school oracy culture on non-verbal reasoning we 
would have to show that the Raven's age standardised scores had increased over time and this was 
not the case. 


To statistically evaluate the impact of the oracy curriculum for the intervention school an analysis of 
covariance was undertaken with the school (intervention vs control) as the between-participants factor 
and raw scores on the Raven’s Matrices as the dependent variable. The pre-intervention Raven's raw 
scores were entered as the covariate. The analysis showed no significant difference between the two 
schools post-intervention once the pre-intervention scores had been partialled from the analysis 
(F(1,35) = 0.96, p = 0. 33, p2 = 0.03). As we would expect there was a significant relationship 
between pre-intervention and post-intervention scores (F(1,35) = 35.95, p < .001, 7p2 = 0.51). A 
similar pattern was seen in the analyses of the age standardised scores on the Raven's Matrices 
showing that the difference between the schools’ post-intervention with the pre-intervention scores 
partialled out was non-significant (F(1,35) = 1.23, p = .276, » p2 = 0.03). These analyses indicate no 
particular benefit of the oracy curriculum utilised by School 21 in terms of non-verbal reasoning as 
measured by the Raven's Progressive Matrices. 
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Table 3.1: Means and standard deviations (in parenthesis) for the students from intervention 
and control schools for the Raven's Progressive Matrices” 

Pre-intervention Post-intervention 
Raw scores Raw scores 


Standard 
scores 


Standard 
scores 
Intervention 31.25 
school (n = 20) (6.28) 


Control school (n 33.06 
= 18) (3.95) 


We should note that absence of an improvement in the Raven's scores should not be seen as 
evidence of an absence of impact of the oracy intervention but rather that there does not seem to be 
an impact on the non-verbal reasoning skills as measured by the Raven's Progressive Matrices. 
Additionally, this could indicate a lack of power in these analyses given the rather small sample size. 


In summary the pre and post Raven's analyses show that there appears to be no benefit from the 
oracy curriculum on students’ non-verbal reasoning skills. This suggests that the curriculum may not 
have more general benefits for student attainment beyond improvement in their language skills. 


Pre- and post- test using the prototype University of Cambridge Assessment Toolkit 


In this section we present the results of the assessments conducted by teachers with 12 School 21 
students and 12 students in a comparison school using the prototype Oracy Assessment Toolkit. 
These analyses, conducted by the University of Cambridge team and reproduced here, are set out in 
Mercer et al. (2014: 23-26). The evaluation team were not involved in these assessments. 


In their first analysis the Cambridge team compared teachers' ratings of the three assessment tasks 
at the initial (October 2013) and follow-up (April 2014). For School 21 these indicated a very small 
amount of student progress. The difference between the initial and follow-up task means was +0.47 
for the individual presentations (initial mean = 3.64, follow-up = 4.11), +0.80 for the group talking 
points task (initial mean = 4.20, follow-up = 5.00) and -0.78 for the paired instructional task (initial 
mean = 4.61, follow-up = 3.83). Likewise, their scores indicate little student progress at the 
comparison school where the differences between initial and follow-up means were: individual 
presentations = -0.23 (initial mean = 4.46, follow-up = 4.23); group talking points task = -0.48 (initial 
mean = 4.50, follow-up = 4.02); paired instructional task = +0.25 (initial mean = 4.75, follow-up = 
5.00). 


The Cambridge team considered that the low progress may be accounted for by high initial ratings by 
the teachers who knew the students that they were rating. They therefore sought to eliminate the 
potential bias by undertaking researcher assessments of the videos of the students undertaking the 
assessment tasks. These assessments were undertaken by the University of Cambridge research 
team who were very familiar with the ratings scheme. The results for the researcher ratings are 
shown in Tables 3.2 to 3.4 below: 


Table 3.2: Individual presentation task mean ratings by University of Cambridge researchers 
School 21 Comparison School 
Initial task 


Follow-up task 
Difference in means 


2 The means included here are only for those students who completed the Raven's Progressive Matrices at both pre- 
intervention and post-intervention testing phases. 
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Table 3.3 Group talking points task mean ratings by University of Cambridge researchers 
| ESTod aTolo) pak Comparison School 


Initial task 4.17 2.33 
Follow-up task 5,25 3.00 
| Difference in means 0.67 


Table 3.4: Paired instructional task mean ratings by University of Cambridge researchers 
ESTod aTolo) Iai Comparison School 


Initial task 


Follow-up task 
Difference in means 


These assessments provide a clear indication of progress at School 21 in relation to the individual 
presentation task and the group talking points task (difference in means +1.64 and +1.08 
respectively). There is a less clear, but positive, indication of progress at School 21 in relation to the 
paired instructional task. As the Cambridge team point out, the paired instructional task was changed 
between the initial and follow-up test as some students misunderstood the initial task, so less 
confidence can be placed on the findings from this task. 


In comparing the results from School 21 to those from the comparison school at the initial test, the 
mean ratings for group talking points and the paired instructions task were higher in School 21 than 
the comparison school, and for the individual presentation the mean ratings at School 21 were lower 
than the comparison school. The Cambridge team suggest that the noticeably higher initial ratings for 
School 21 students on the group and paired tasks may reflect the fact that by the time the initial tasks 
were conducted in October the students had already been introduced to the oracy-led curriculum. The 
evaluation team would agree with this suggestion. There is no evidence to suggest that the School 21 
pupils had higher academic ability at this stage than the comparison school; in fact our analysis of the 
Raven's Matrices suggests if anything the reverse. 


The ratings for School 21 were higher for all the follow-up assessments than those of the comparison 
school where rates of progress were lower than for School 21. Overall the data indicate that for each 
task students’ progress was greater in School 21 than in the comparison school. The Cambridge team 
did not have sufficient data to test whether or not the difference was statistically significant and 
therefore these conclusions should be treated with a good deal of caution. 


As we outlined in Section 2, there are some important limitations in using this prototype Oracy 
Assessment Toolkit to attempt to measure impact at a development stage in a project with a small 
sample of students in one school. While the Cambridge team claim that the results can 'be cautiously 
taken as an indication that School 21's oracy-led curriculum is making an impact on the development 
of children's oracy skills' (Mercer et al., 2014: 25-26), testing of a larger group of pupils would be 
necessary to substantiate impact claims. 


It is interesting to note that the assessments indicate that students made the most progress in relation 
to the presentational task. This reflects a key finding from our qualitative work that the School 21 
oracy curriculum implemented at School 21 has a particularly strong emphasis on presentational talk. 
It also supports the explanation given above for the lack of progress found using the Raven's Matrices 
test, which is a stronger proxy measure of the development of exploratory rather than presentational 
talk. 
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3.2 Impact on student engagement 


Focus groups with students and classroom observations suggested that there is strong student 
engagement with the dedicated Year 7 oracy curriculum and with using talk to support learning across 
the curriculum. This seems to be associated with: 


e lessons couched in ‘real’ contexts and meaningful tasks 

e children’s experiences/perspectives and existing uses of talk are used as the starting point 

e clear routines for managing/organising oracy (e.g. coaching groups, protocols for group work) 
e shared frameworks for talk (feedback on ‘desirable’ oracy behaviours; sentence starters) 

e the role of coaches in encouraging engagement in the collaborative learning process 

e the establishment of ‘safe’ teaching and learning spaces where many are confident to talk 

e opportunities for students to make choices about how to use oracy protocols. 


In lessons observed there were multiple opportunities for students to engage in discussion in 
groupings of varying size. Most students readily engaged in discussion and routinely used the oracy 
techniques they had been taught, e.g. turn-taking, building on previous speaker, use of sentence 
stems, within group sessions. Teachers who had worked in other schools felt that students at School 
21 had greater confidence to speak in different circumstances than others they had taught. One, for 
example, felt that the explicit opportunities to present their ideas and views led to higher ‘levels of 
eloquence’ (Visit 1 Teacher Focus Group), and another commented that students at School 21, 


are more confident and willing to express their ideas and not that bothered by being wrong, in 
my last school they were just afraid. Here there’s like 17 hands go up, in my old school I'd be 
the one trying to gee everyone up to speak but here it’s the opposite’ (Teacher interview Visit 
3A) 


Staff we spoke to at School 21 were also convinced that the focus on oracy is related positively to 
learning outcomes across the curriculum. The project director told us that students... 


...enter below national expectations and we think our data is saying that by the end of Year 7, 
they’re above... You can’t disaggregate the success of the students in their maths for example 
from their talk; progress in maths is well above national average in similar schools and the 
same in English, and | think that’s entirely down to the ability of the children to talk about and 
reflect on their thinking. 


When asked to explain why this may be the case, he focused on the role of talk for learning: 


That came across in the maths assessment and it became very obvious that they were used 
to using talk protocols to solve problems mathematically and how they went about tackling it 
definitely relying on paired work to do that, and a lot of subjects would emphasise the 
importance of talk, of working as a group in order to solve a problem, talk can help you, the 
sort of inter-thinking, the usefulness of talking to other people, sharing ideas. 
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4. Findings: Oracy Skills Framework 


4.1 Introduction 


In this section, we begin by considering how well the Oracy Skills Framework (Appendix 1) represents 
the skills required by children and young people to succeed in education, employment and life. We 
then report our findings on the application of the framework in School 21 and its effectiveness as a 
diagnostic tool to identify students' strengths and areas for development and for supporting 
progression in oracy. 


4.2 Structure of framework: The four strands 


The Oracy Skills Framework presents oracy development in terms of four strands of skills: cognitive; 
physical; social and emotional; and linguistic. The framework appears to draw on a range of 
traditions of research related to language and meaning and their role in social interaction, although 
these are not made explicit. It is recognised that there is overlap between the four strands. However, 
as the framework is designed to be ‘teacher-friendly’, it has been designed to be accessible and not 
overwhelming. As a member of the Cambridge team explained: 


There's always this tension between what teachers might genuinely use because it won't take 
weeks of time to prepare or do and covering things in a totally comprehensive way. 


The team expand on this in their final report 


Although we have striven to make the framework as comprehensive and accurate as 
possible, our aim of creating something that would be accessible and useful to practising 
teachers meant that we had to balance detailed accuracy and complexity with clarity and 
practical usefulness. We therefore readily admit that our framework might not satisfy the 
rigorous criteria of an academic linguist; but our experience in its development and trialling 
encourages us to believe that it will help a teacher perceive more clearly quite what aspects 
of skill are involved in being an effective user of the spoken language. (Mercer et al., 2014: 
11) 


The University of Cambridge have reviewed the framework with an expert advisory panel who agreed 
that its scope was appropriate. 


The focus on skills is distinct from that adopted by other frameworks for curriculum and assessment in 
speaking and listening which have tended to focus on the range of contexts for talk, rather than skills 
that might be applied across contexts. It is not possible to gauge how effective this skills-led approach 
has been in impacting on oracy at School 21 as it is not possible to disassociate this from other 
dimensions of the provision at the school, e.g. the high value placed on talk, the provision of multiple 
contexts for talk, and the focus on reflection on appropriateness of talk to context. However the focus 
on skills, and the division of skills into these four strands, does appear to have been helpful to both 
staff and students at School 21 who, during interviews and focus groups, articulated their 
understanding of the different strands. This shared understanding of the framework is cited by the 
school as a key dimension of their approach as it supports ongoing reflection by staff and students: 


A full understanding of the Oracy Framework then provides the teacher and student with a 
shared language with which to critique their performance and focus in on the skills in a very 
specific way. (Fidoe, 2014) 


In relation to the cognitive dimension, School 21 teachers described their commitment to a 
‘constructivist group work approach to learning’. As the head teacher explained, 
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. obviously a talking classroom produces higher order thinking but you've got to see 
exploratory talk as a pedagogy, a way of thinking and a way of producing better thinking. 


In terms of the social and emotional dimension, the curriculum is underpinned by a firm 
commitment to enabling students to ‘find their voice’ and this is seen as essential to confidence and 
future participation: 


Head teacher: By giving students the chance to find their voice and having the chance to 
speak in a range of settings you are making them not just prepare for their life that they will 
lead but also it gives them a sense of confidence and wellbeing that they can articulate and 
use and be listened to. 


The emphasis on oracy is seen as particularly empowering for those students who find it difficult to 
express themselves and explore ideas through writing. During a classroom observation (Observation 
Visit 3B), one student gave an impromptu performance of his ‘Ignite’ speech®. The teacher explained 
afterwards that this student is dyslexic and has difficulty writing, but that he can engage orally with 
difficult ideas and present these. Two students commented: 


In a group you socialise which helps you in the future because you can work with anyone 
whether you like them or not you'll be able to work with them without fussing or complaining. 
(Visit 2, Pupil focus group) 


In oracy we have to speak in front of everyone which makes us like I'm not really nervous 
now, but like he said we have to speak and it just makes you feel better. (Visit 3, Pupil focus 


group) 


The physical dimension is seen in terms of aspects such as tone, pitch, facial expression, gesture, 
posture and eye contact. The school’s strong emphasis on drama has provided rich opportunities to 
develop the physical dimension of talk for presentational purposes. As one student noted: 


In my oracy session with Mr xxx we discussed about being ‘strong and wrong' so if you get 
something wrong you can just carry on, don't like pause in the middle. (Visit 2, Pupil focus 


group) 


The linguistic dimension has been a particular focus for development in 2013/14 informed mainly by 
consultancy with Wendy Lee of The Communication Trust. During the period of the evaluation, 
teachers at School 21 were exploring ways of supporting vocabulary development, for example by 
differentiating between three tiers of language (Beck, 2013). As the project lead explained this is... 


...based on the principle that there are 3 tiers of vocabulary: tier 1, 2 and 3. Tier 1 is very 
basic words. Tier 3 is very specific words like photosynthesis...technical words. Those tier 1 
and 3 words tend to get taught explicitly but tier 2 are often not, they are words like discuss, 
reflect, compare; the sorts of words you find in broadsheet newspapers rather than tabloids 
and now we've got a programme to teach those words explicitly. (Telephone interview with 
project lead) 


Students commented on how feedback had helped them revise the language they used: 


In that first week we were looking at other Ignite speeches to motivate us to do our own and 
Miss xxx told us we have to use new vocabulary and be eloquent and also to have a good title 


3 Students in Year 7 all develop and present ‘Ignite’ speeches based on the format developed through Ignite events 
(http://igniteshow.com/), during which presenters talk for 5 minutes about a subject of personal or professional interest using 20 
slides that auto-advance every 15 seconds. See School 21 examples at http://school21.org/secondary/beautiful-work/ignite- 
speeches-2013. 
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because the title catches the person’s attention, so if you have a good title then they're going 
to like the speech. (Visit 2, Pupil focus group) 


Like the teachers gave us the confidence to perform it, they would just give us advice and 
how you would be able to perform it properly, in a way that the audience would engage with 
you. (Visit 3, Pupil focus group) 


The linguistic dimension is also being developed through discussion about texts (using reciprocal 
reading), a focus on etymology, reflection on use of vocabulary, and an emphasis on choosing the 
most appropriate vocabulary for the context. 


4.3 Application of the framework 


Given that all four strands could be seen as applicable to any instance of talk, it is possible they may 
be difficult to untangle in practice. Distinguishing between different skills for teaching and assessment 
purposes could lead to an atomistic curriculum which artificially distinguished between different skills 
in unhelpful ways, for example through narrowly framed activities. However, our fieldwork suggests 
that this does not appear to have happened at School 21. It appears that the framework has been 
valuable to the school in helping to articulate the scope and range of oracy, and that this focus has 
not distracted from the need to plan for meaningful contexts for oracy development. 


The framework is given high profile within the school. Classrooms display notices related to giving 
‘proof of listening’, ‘professional behaviour’, ‘volume of talk’, ‘working hard and showing respect’, class 
rules, lists of vocabulary and types of talk (e.g. explaining, describing, explaining, comparing, 
justifying, predicting, creating, etc): 


We've got various oracy displays up to show those different categories visually. One of them 
is a graffiti display where the children can jot on different ways of recognising the different 
areas and one of them is just quite an informative display. (Telephone interview, Oracy 
teacher) 


4.4 Diagnostic use of framework 


At School 21, the 4-strand framework appears to work on a number of levels. It provides a starting 
point for staff development, school review and to analyse the needs of a particular cohort. For 
example, School 21 told us they initially focused on the cognitive strand of the framework but teachers 
identified a need to develop the physical strand (pitch, eye contact, etc) and this led to a focus on 
performance poetry. As explored above, and following further review, the school is now strengthening 
provision for the linguistic element: 


| guess the progression in oracy highlighted to us that [the linguistic strand] was a particular 
area we needed to work on and we started to bring it in with the extra assemblies, some 
coaching group work on vocabulary and some targeted vocabulary work in the oracy lessons. 
(Telephone interview, Oracy teacher) 


The four strands are also used diagnostically to identify individual students’ strengths and areas for 
development. The framework is also used explicitly with the students to support self and peer 
assessment. The framework also informs teacher, peer and self-assessment. As this teacher went on 
to explain: 


We've tried to make the 4 strands permeate every aspect of the work, when they are drafting 
[the Ignite speech], redrafting it, then they are coaching each other [...]the children are able to 
explicitly say the area that they think they are strongest in and the area that they think they're 
weakest in. (Telephone interview, Oracy teacher) 


Education Endowment Foundation 25 


Oracy Curriculum, Culture and Assessment Toolkit 


There is a recognition that different students will find different contexts more or less challenging, and 
that school cannot prepare students for all contexts they might encounter. The framework is used to 
inform approaches that engage students in considering what might be appropriate in different 
contexts. As the project manager commented: 


The nature of the situation is a skill in itself ... it's to do with register and genre, ... if you teach 
specific contexts like you say here are the appropriate phrases for a job interview or for x, y 
and z then you're not really teaching them oracy, you're teaching them how to finish school. 
So that's why we've basically suffused it throughout the skills framework, it's always about 
context and the problem with that is there is an infinite number of contexts. (Visit 2 Interview 
with project manager) 


4.5 Using the framework to support progression in oracy 


The framework is designed to support a recursive curriculum, revisiting different skills in relation to 
increasingly challenging content and contexts, where context is understood in terms of the purpose 
for talk, and the type and size of audience. For example, the Year 7 Ignite talk is based on a self- 
selected topic. In preparation for this, children are required to present their talk in what might be 
expected to be increasingly demanding contexts, so children move from presenting to their peers, to 
students in other classes, to a public audience. In Year 8 students are required to create and present 
a second Ignite talk. This second ‘Expertise Ignite’ involves creating a speech based on an essay. 
This is more demanding as it requires students to engage with unfamiliar material. One of the 
teachers responsible for developing the original framework explained that ‘if Year 7 is finding your 
voice, Year 8 is about developing the voice’. As the project manager explained: 


In Year 7 you're already in quite a challenging situation because you've come to a whole new 
school and you're dealing with ... unfamiliar people in different sized groups, some of them 
you know better than others, you're not entirely sure how formal the whole situation is, you 
don't have much in common with them ... that's a tricky context, but by the time that you're in 
Year 8 you've got friends so you need a new challenge and so it's like we'll give you a harder 
purpose, to persuade someone of your argument because that's harder than just describing 
something and we'll give you a large audience of adult strangers because that's more difficult 
than a smaller audience of familiar teenagers. (Visit 2 Interview with project manager) 


The curriculum is designed to support a developing awareness of context so that students can make 
appropriate choices about their use of talk. 


4.6 Summary and recommendations 


The framework seems well designed to inform the development of an oracy curriculum that will 
support progress in oracy. At School 21 the framework appears to have supported staff to develop a 
broad-ranging oracy curriculum for their students to provide a variety of purposeful contexts for talk. It 
provides the basis for a shared language for oracy teaching across the school. School 21 has used 
the framework diagnostically to inform staff development, the focus of teaching within different units 
and to support the progress of individual students. This demonstrates how the framework can be used 
productively by specialist and non-specialist teachers to inform curriculum development and support 
progress in oracy. 


The framework's effectiveness however will depend on how it is interpreted and operationalised. It is 
not intended to determine a prescriptive or atomised curriculum, but rather to be used diagnostically 
to inform the focus and direction of teaching and learning activities. Whereas School 21, for example, 
highlighted the linguistic dimension as an area for development with their students, other schools may 
highlight other areas or, perhaps more likely, the focus may differ in the context of different audiences 
and purposes for talk. It is therefore essential that any guidance for other schools acknowledges this 
flexibility, and supports appropriate choices about the direction and emphasis of oracy provision in 
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different school contexts. Moreover, in foregrounding the framework in presenting the curriculum, 
there is a risk that School 21’s emphasis on providing diverse contexts for oracy could be missed. 
(This is explored further in Section 5.2.) It is also important to emphasise that the four strands of the 
Oracy Skills Framework — cognitive, social and emotional, physical, and linguistic — are closely related 
in multiple ways. Careful explication is necessary to avoid the kind of decontextualised or atomised 
teaching that may not be conducive to oracy development. 


We recommend that: 
Guidance for other schools needs to emphasise that: 


e the four strands of the Oracy Skills framework are closely related and to be effective teaching 
needs to be contextualised and should not overly focus on individual skills in isolation from 
other skills 

e the framework should be used flexibly to support appropriate choices about the direction and 
emphasis of oracy provision in different school contexts. 
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5. Findings: Dedicated Year 7 curriculum 


5.1 Introduction 


During the pilot period School 21 implemented and further developed the dedicated Year 7 curriculum 
in weekly oracy lessons, oracy in every lesson and a whole school oracy culture. These three 
components of the Oracy Curriculum, Culture and Assessment Toolkit are highly related. For example 
oracy techniques taught to pupils in the dedicated oracy lessons are used by other teachers in subject 
teaching and in whole school activity such as assemblies. In order to report our findings clearly we 
have reported our findings on each of these components separately: in this section we review the 
dedicated Year 7 oracy curriculum, in Section 6 we review the approaches to oracy in every lesson 
and in Section 7 the strategies and techniques used to build a whole school oracy culture. 


It is therefore not possible to arrive at definitive judgements about its appropriateness or the 
effectiveness of the three components, reviewed in Sections 5-7, either in terms of scope or ease and 
effectiveness of application, because they were still being created and/or refined during the pilot 
period. It must be emphasised that further development work is still ongoing and that areas that we 
identify for development may now have been addressed. They are highlighted here however as they 
are areas which we suggest will be important foci for other schools adopting the approaches to 
consider. 


The findings in Sections 5-7 are based on analysis of our school visits staged over the pilot period 
which comprised discussion with staff and students, and observations of oracy lessons, other subject 
lessons and whole school activity. In our commentary we at times distinguish between the 
intervention as described on the Voice 21 website (which was created at the end of the project) and 
the enactment of the intervention in School 21. 


In this section we focus on the dedicated Year 7 curriculum. 
5.2 Dedicated Year 7 curriculum : Overview 


The Year 7 oracy curriculum includes four units: Finding your voice, Performance Poetry, Persuasive 
Techniques, and Ignite. Each unit is designed to provide a variety of contexts in which students can 
develop, use and apply the skills represented by all four strands of the Oracy Skills Framework. Units 
are structured in three phases: immersion, skills development, and performance; and are supported 
by a series of lesson plans and resources that provide guidance for less experienced teachers but 
that could be adapted as necessary. The first unit, Finding your voice, supports students in reflecting 
on their own use of talk and also introduces a range of protocols that will be used across the 
curriculum. Subsequent units are designed to consolidate skills explored through preceding units. 


On the Voice 21 website, an overview of each phase is provided along with lesson plans and a series 
of resources. Some units are more developed than others and some resources are currently provided 
with little or no commentary. Some further narrative is needed to indicate how the units are intended 
to be used, i.e. whether they are intended to constitute a coherent series of lessons or whether they 
are sample lessons and/or how they might be adapted. Lessons are presented in different formats 
and with different levels of detail, e.g. some as PowerPoints and some as Word documents. This 
means that sometimes the stages of the lesson are made explicit while at others they are implied. 
Some approaches are currently included as handouts, e.g. ‘Group Protocols’, with no accompanying 
lesson plans to indicate how these have been introduced or contextualised in practice. Some phases, 
e.g. Phase 3 in Unit One, do not yet have any lesson plans. It would be helpful to agree a consistent 
format for the level of detail that needs to be provided for each lesson. This would be likely to include 
statements of the key opportunities and objectives of the unit as a whole and of each lesson, possibly 
along with suggestions of possible adaptations and contexts for learning. 
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The decision to examine talk in depth through four units seems highly appropriate as it allows 
sustained engagement with particular types of talk and greater potential for children to develop 
confidence and skills over time. The disadvantage of this approach is that the curriculum could be 
seen as neglecting broader repertoires of talk. The emphasis on developing talk across the curriculum 
is therefore important. 


During visits to School 21, it was clear that the talk curriculum took place within a range of contexts 
that were meaningful and relevant to the students. In our professional opinion, this approach is a 
strong feature of the school’s provision. However this dimension, which would be seen as an essential 
feature of an effective talk curriculum, is not always highlighted within the lesson plans. While it is 
recognised that the framework focuses on skills, there is a need to emphasise that these skills need 
to be addressed within diverse contexts, and that talk needs to be evaluated in relation to 
appropriateness to context. We consider aspects of the dedicated Year 7 curriculum in more detail 
below. 


5.3 Talk for presentational purposes 


The curriculum as implemented at School 21 includes a strong emphasis on talk for presentational 
purposes and the curriculum as outlined on the Voice 21 website reflects this in its guidelines for 
supporting talk in formal contexts and talk for persuasive and presentational purposes, e.g. through 
formal debate, pitching an idea and Ignite presentations. This dimension draws heavily from drama 
practice. Support for developing talk for formal presentational purposes appears to be a strength of 
the curriculum. Indeed, it was noticeable that students tended to focus on such opportunities when we 
spoke to them about oracy. During student focus groups, students spoke confidently about the 
different strands of the framework but did so primarily in terms of presentational talk. When asked to 
explain the strands, for example, they identified a need to focus on ‘physical, emotional and social’ 
dimensions, and then qualified these with the following comments: 


e Physical is when you start using hand gestures and show body language to the audience and 
it's another way to interpret what you're saying as well. 

e Emotional is where your words have like meaning so it's going to get the audience fired up 
and make them think something else. 

e One! think is called cognitive and that is like when you're speaking you're thinking of what to 
say next. (Comments from Visit 2 Pupil focus group) 


The focus on developing presentations appears to be a real strength of the school’s approach. During 
lesson observations, aspects of spoken language were taught explicitly, e.g. tonal shift, and students 
were able to make use of these in their presentations. Student comments, for example, suggested 
their engagement with the Ignite unit of work had impacted on their ability to analyse and review their 
talk for presentational purposes. One teacher commented that some students may overuse certain 
elements. Making careful judgements about how to draw on different skills for different audiences will 
be an area for development for the school. The focus on ongoing reflection and review seems well 
designed to support teachers and students in deciding which kinds of rhetorical strategies are 
appropriate in any given context. We encountered less evidence of students reflecting in depth on 
their use of talk for learning, and this is explored below. 


5.4 Talk for learning 


As explored in Section 4.2, School 21 state a firm commitment to promoting talk for learning and there 
was certainly evidence of this in our fieldwork. Teachers spoke of the value of group talk, for example, 
and we saw many examples of opportunities for discussion and of teachers providing support for 
students to engage in group discussion. However, at the time of writing, the curriculum as presented 
on the Voice 21 website includes less focus on this dimension of oracy. Many opportunities for group 
discussion are provided but there is less emphasis on what effective group discussion looks like or 
how it might be supported. In the ‘Four Coaching Strands’ unit, for example, the cognitive strand is 
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defined in terms of ‘clear arguments’ and ‘good organisation of points’. While these are elements 
worth exploring, this suggests that the cognitive strand is being understood in terms of the 
communication of knowledge with less emphasis on exploratory talk. 


It is our professional view that the curriculum as outlined on the Voice 21 website needs to include 
greater emphasis on the different kinds of talk that might be appropriate in different contexts and for 
different purposes, to include a focus on the kind of talk that might be associated with developing 
thinking or understanding. For example, in Unit One, the same prompts are used to discuss the 
nature of a group discussion and a ‘just-a-minute’ presentation (See Oracy Critique lesson and Oracy 
Analysis Practice session). The kind of talk you might expect from these two tasks is different. It may 
well be that these differences emerge in the discussions following the children’s review, and the 
materials do refer to differences between formal and informal talk, but this is not made explicit in the 
commentary. Similarly, distinctions between ‘good’ talk and ‘bad’ talk are made (e.g. see lesson 
Talking about Talk). This may be misleading as it suggests there is one type of good talk. 


5.5 Reflection on talk 


The approaches used by School 21 provide strong support for developing a shared language to 
support organising for and reflecting on talk. There is a focus on making talk explicit, working with 
students for example to generate lists of what makes effective talk, and encouraging students to 
review others' talk, for example that of visiting speakers. This includes considering body language, 
turn taking, and group dynamics. As one teacher commented, 


One of the last lessons we did working in groups of three, they did a little pie chart of who'd 
contributed in a three ... getting them thinking about how much they contribute when they 
contribute, do they have to contribute the same amount each time or not ... and just that 
awareness there of ‘these are the roles | have to play, these are things that | have to be 
aware of’ and so this term I'm giving them the chance to show that they've understood that. 
(Visit 2 interview, oracy teacher) 


There is also a strong emphasis on encouraging students to reflect on their use of language in ways 
that will position them strongly in relation to others, looking forward to employment situations and their 
broader adult life. One teacher explained: 


Year 7 focused on the basics of vocabulary between formal and informal usage, e.g. ‘the 
language of home, language of playground, language of school’. Getting to Year 8 it is about 
taking their vocabulary ‘up a notch' and seeing if they can understand ‘the language of being 
taken seriously’... academic language and if they can think of the right types of situations they 
can use that in. (Telephone interview, oracy teacher) 


As exemplified and explored further in Section 6.4, students we spoke to and observed engaged 
readily in this reflection although comments tended to focus on the physical, social and emotional 
dimensions and less on relationships between talk and understanding or thinking. 


Vocabulary development has been a particular focus for the school. The school has trialled a range of 
approaches. The most effective approaches are likely to be those that teach vocabulary development 
in contextualised ways. The same teacher provided examples of this: 


So in the role of an expert they need to use precise, technical language that's connected to 
what they're speaking about ... the tier 2 words being things that could cut across different 
subjects, things like analyse, summarise, profile words like that that connect to broadsheet 
reading. (Telephone interview, oracy teacher) 


School 21 are also implementing a vocabulary programme they have devised with Wendy Lee of the 
Communications Trust and which is currently being rolled out to schools in East London. School 21 
state that initial feedback from schools suggests that this approach ‘is having a significant impact on 
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their students; linguistic competency’. At this point, it is not clear whether or how this is being 
integrated within the dedicated Year 7 oracy curriculum. 


5.6 Summary and recommendations 


Given the developmental nature of the project we are unable to arrive at a definitive judgement about 
the efficacy of the dedicated Year 7 oracy curriculum. For example, during the evaluation period, the 
curriculum was undergoing ongoing review with a particular focus on vocabulary development and 
further refinements may take place as a result of this. 


The four units that comprise the proposed curriculum appear to be well selected to explore the four 
strands of oracy skills in the Oracy Skills Framework and to establish strategies that can be used to 
support learning across the curriculum. The distinctive quality of the dedicated Year 7 curriculum is 
the identification of a set of skills which may be used across different contexts, with an emphasis on 
making appropriate choices about how to draw on these appropriately within these different contexts. 
Each topic is examined in depth and each unit culminates in a motivating opportunity for application of 
skills learned. 


Provision for developing talk for presentational purposes appears to be a particular strength and the 
work draws effectively from drama practice. There is an emphasis on the effective communication of 
an argument or point of view in formal and informal contexts and this seems to be directly aimed at 
preparing students for economic success and participation in civic life. The curriculum guidance 
available on the Voice 21 website includes less detail on: the development of talk for learning; the 
need to provide informal opportunities for talk; and the need to embed the oracy curriculum within 
diverse meaningful and motivating contexts associated with different kinds of talk. 


In any process of curriculum design, decisions are made about what to include and some possible 
elements are excluded. For example, the school has chosen to focus in depth on selected 
opportunities for talk which appear to be well chosen. However, there is a need to be more explicit 
about how contexts might be provided across the curriculum to ensure students develop broad 
repertoires of talk. 


It is worth noting that previous attempts to develop provision for oracy have generated other 
proposals which perhaps reflect different values and priorities (e.g. see Norman, 1992; QCA, 2003). 
An oracy curriculum might for example: explore metalinguistic awareness through considering 
different languages and dialects; include a focus on oral heritage; or provide opportunities for talk that 
are more conversational and less structured. It may be that the School 21 curriculum develops to 
include some of these elements. In any case, however, there is a need to recognise that the School 
21 curriculum is underpinned by particular aims, objectives and principles, and these need to be 
stated clearly. For example, if the curriculum is designed primarily as a way of supporting talk in 
formal contexts then this needs to be stated. 


It is our professional view, however, that the oracy provision at School 21 is, and should be, more 
inclusive than this apparent emphasis in the materials suggests. The curriculum, as presented on the 
Voice 21 website, therefore needs to include a clearer statement about the different kinds of 
opportunities that should be provided to support oracy. This would require explication of the value of 
both formal and informal opportunities for talk, and support for schools in providing opportunities for 
students to use talk for learning, i.e. to use talk to develop ideas as well as communicate them. 
Addressing this will involve focusing in more depth on the cognitive strand of the framework. 


We recommend that: 


School 21 continues to refer to expertise and approaches developed outside the school to ensure that 
provision takes account of recent research studies and development projects. This would include 
revisiting support for promoting talk for learning, particularly in relation to exploratory talk. 
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The dedicated Year 7 curriculum documentation and associated resources be further refined in terms 
of content and presentation. In particular these should: 


e clearly state the aims, objectives and underpinning principles of the dedicated Year 7 
curriculum so that other schools are clear about what the curriculum is, and is not, designed 
to achieve 

e highlight the role, nature and development of exploratory talk 

e emphasise the need to provide a range of motivating and relevant contexts for talk. 
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6. Findings: Oracy in every lesson 


6.1 Introduction 


In our professional view, a particular strength of the School 21 approach is the emphasis on ensuring 
that approaches and skills developed through oracy lessons are applied across the whole curriculum. 
In this section we explore how this cross-curricular approach is implemented, identify strengths that 
can usefully be shared with other schools, and make some recommendations for further development. 
In doing so, we refer to the resources available on the Voice 21 website at the time of writing, and to 
approaches we observed during visits that could usefully inform the future development of those 
resources. 


6.2 Approaches used across the curriculum 


Selected approaches are included on the Voice 21 website with recommendations for their use across 
the curriculum, e.g. related to groupings, protocols, and references to three tiers of language. It is 
worth noting that the provision for oracy at School 21 has also drawn on a variety of other approaches 
and techniques for supporting collaborative talk for learning: e.g. The Kagan Method 
(http://www.kagan-uk.co.uk/) and CASE (Commission of Accelerated Science [Education 
http://www.kcl.ac.uk/sspp/departments/education/research/crestem/Research/Past- 
Projects/Cognaccel.aspx) described by one of the teachers as 'a way of thinking and talking about 
science which is very logical and cognitive and the lessons are brilliant’. 


Teachers we spoke with told us that students now expect to use talk for learning as they become 
more used to these approaches. For example, one teacher noted: 


You have some high ability kids and some low ones, so get them to explain what they did. | 
think we promote it so much and we're OK with the talking, | think it’s embedded in them so 
much now, it’s how they approach every task they do, they know straight away they can talk 
about it without us saying ‘you shouldn't be talking about this’. (Visit 3 Teacher focus group) 


6.3 Organising for talk 


The school has introduced a number of protocols to provide support for different kinds of group 
discussion. Students commented on how the protocols supported the management of group work: 


When you're in the groups you're actually getting more ideas and you're saying your ideas 
and everyone can hear you share your ideas. 


When you're in a group you can share your knowledge and your ideas ... and when you 
negotiate with people in the group, whether it's six people or three, in that amount of time you 
still get loads of stuff from it. (Visit 2 Pupil focus group) 


These protocols are well embedded and in use across the curriculum. Teaching approaches 
developed in oracy lessons are used elsewhere across the school curriculum. For example, the 
mathematics teacher often uses the ‘traverse formation’ (observed in a drama lesson where students 
stand in two lines facing each other) for mental arithmetic as it means that students rely on talk to 
solve problems. This consistent approach seems to be having an impact on how students approach 
learning. During our visits, students were observed readily introducing protocols during group work, 
standing to present their ideas, using sentence stems to help structure contributions, and using 
strategies for turn-taking and presenting orally. 
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6.4 Talk for learning 


Creating a climate in which talk is valued would certainly seem to be an important condition for 
exploratory talk to thrive. A focus on the role of talk for learning underpins the school philosophy and 
is reflected in approaches to classroom organisation, curriculum development, pedagogy (e.g. 
problem-based learning) and continuing professional development for staff. Despite the reservations 
expressed in Section 5.4 about the oracy curriculum as represented on the website, teacher focus 
groups and lesson observations suggested that students were applying what they learned in oracy 
lessons across the curriculum in ways that were likely to support learning. They used many of the 
protocols designed to support effective group work. For example, in a mathematics lesson: 


Students are given 3 different shapes made of same number of squares. They are asked to 
discuss in groups what is the same and what is different about these shapes. The group | 
observe starts talking about the task straight away, beginning by deciding how to tackle it 
together. (Observation Visit 1B) 


Teachers spoke of the relationship between talk and learning in their specific subjects. The history 
teacher for example believed that the focus on oracy was supporting historical skills. When asked 
about the relationship between oracy and history, he said ‘it’s impossible to separate the two’. He saw 
the specific contribution of oracy to history teaching as related to the students’ ability to critique: 


| think the fact that they’re able to talk through their ideas and are starting to understand that 
an argument is better when they have facts and evidence to support it. Now they'll say things 
like ‘what source did you get that from?’ and | don’t think that would come through without the 
oracy programme, | think it has to be a whole school thing — | think if | was pushing that in 
history it wouldn’t work. (Teacher interview Visit 3A) 


Students, however, when asked to define oracy during our focus group, focused primarily on 
presentational aspects, for example: 


Oracy is about like developing your eloquence and also being able to perform to make 
yourself heard and also just developing your voice further whilst using hand gestures and 
different oracy attributes. 


It's also about using your eye contact and getting the audience involved using body language. 
(Visit 2 Pupil focus group) 


While these reflections were useful, the emphasis on presentation did raise questions for the 
evaluation team about the place of exploratory talk. It is quite possible that this emphasis on 
presentation reflected the focus (at the time of visits) of oracy teaching linked to Ignite talks. It is also 
perhaps easier to refer to these aspects of talk than it is to reflect on group discussion. Some students 
did comment on group work, but focused on management of group discussion rather than on how this 
might support the development of ideas or understanding: 


Another skill in oracy is when you're working in groups, like larger than what you'd expect it to 
be because normally you're happy to work in twos but in oracy you have to work in threes or 
sixes, just larger than you might have expected. 


It's also about like manners so for example if someone was talking you would have to show 
them that you're listening so the speaker can feel more comfortable and more in the zone to 
speak aloud to you. (Visit 1 Pupil focus group) 


Classroom observations certainly suggested that students were keen to engage in group discussion 
and all group discussions observed were clearly related to the tasks in which students were engaged. 
On occasion, students were observed engaging comfortably and confidently in discussion and the 
protocols seemed to enable and support this, as exemplified in the following extract from field notes 
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from a lesson observation. In this example, well-established routines for turn-taking seem to support 
students in taking a whole class discussion in a direction unplanned by the teacher: 


During a feedback session the teacher asks a summariser to reflect on their learning. As she 
does so, another student puts up his thumb [a signal that he wants to speak]. It seems that 
what the summariser said had prompted an idea. Seeing the raised thumb, the teacher nods 
for him to speak. As he speaks, others raise their thumbs too and, taking this as a cue, the 
speaker nominates someone to respond. Response follows response and the talk shifts from 
a cumulative series of summaries to an exchange of views during which students build on and 
challenge each other's ideas: 'I agree but | also think that...', ‘| agree but sometimes you 
can...’ Girls and boys are equally represented here. The teacher listens as the discussion 
unfolds [he later tells me he didn’t plan for this discussion] then returns them to the planned 
routine of summarising, until all groups have finished doing so. (Observation Visit 3A) 


In this example, and in others observed, students drew clearly on turn-taking protocols and used 
sentence starters (e.g. ‘| wonder if...’, Alternatively...’) to help structure their contributions. They often 
added to others’ ideas, and sometimes asked one another for clarifications. 


As might be expected during a development phase (for students and staff), this shared language 
seemed to work with varying levels of effect. In some observations, it seemed that protocols and 
sentence starters worked to support formal uses of talk (e.g. for presentations or debate). They may 
have been less supportive of the more free-ranging exploratory talk described in the opening of this 
report. 


It is important to emphasise that we conducted very few observations during this evaluation. It is 
therefore not possible to draw conclusions here about the quality of exploratory talk across the school, 
but it is worth noting that this may be an area to explore further, particularly in the light of comments 
about the curriculum in Section 5.1. 


The strong emphasis on drama appears to provide compelling contexts for talk and for developing 
skills related to the four strands. For example, the drama teacher told us that children often used ‘2 
tier’ language (see page 25 for definition) during drama lessons. As he noted, during drama 'they want 
to say something, they stretch for language’. The emphasis on meaningful contexts for talk aligns with 
a focus on meaningful contexts for learning across the curriculum at the school. For example the 
history teacher reflected on various ways in which talk had been integrated within enquiry-led project 
work: 


We're trying to make real-life outcomes for all our students, so we’re doing a walking tour of 
London — looking at objects of historical significance — choose an object and write an essay 
defending the choice of object and then symbolise this through art. 


Last week we had a Harkness debate“ about why history is often about kings and queens and 
not ordinary people — and the plan is to do another in a fortnight’s time where they defend 
their object — an exhibition event where they stand by their object and talk about it. We tried it 
with Year 8 with a World War 2 exhibition and found that when they had the real audience 
they just stepped up. (Teacher interview Visit 3A) 


During lesson observations, students were observed considering purpose and audience whether or 
not there were prompts from their teacher to do this. This teacher emphasised the importance of 
being able to explain what they had learned and communicate their point of view: 


4 Approach designed to disrupt conventional ‘teacher-led’ discussion by using round-table discussions to enable Socratic 
dialogue between teachers and pupils. 


Education Endowment Foundation 35 


Oracy Curriculum, Culture and Assessment Toolkit 


We use all the protocols to generate fantastic writing, and to talk about reading as well...but 
more profound than that — we had a WW1 exhibition, an art/humanities collaboration, project- 
based, immersing the children in WW1 and producing an installation. A key part of the end 
project is showing their parents around the exhibition. That all came together because of their 
ability to summarise information that’s in front of them and both demonstration of learning and 
stretching of them cognitively, so it’s sort of hard-wired into the curriculum for next year. 
(Teacher focus group Visit 3) 


6.5 Summary and recommendations 


Teacher interviews and lesson observations suggested that the commitment to promote oracy across 
the curriculum in every lesson is a strong feature of the School 21 approach. In talking with us, and 
during lessons, staff and students utilised a shared language for oracy and were familiar with a range 
of approaches for organising, promoting and reviewing talk. The emphasis on providing meaningful 
contexts for talk across the curriculum is a particular strength. 


School 21 attribute student attainment to this strong oracy focus and students appear to expect to be 
involved in group discussion and apply the protocols and skills developed through the dedicated Year 
7 oracy curriculum across all subjects. The evaluation team observed many students drawing on 
sentence stems and protocols to enable their participation in discussion and to structure exchange of 
views. While students appear keen to engage in discussion and debate the evaluation team, we’re not 
able to draw conclusions about the quality of exploratory talk (as we were only able to conduct a 
small number of observations). It would seem that the development of exploratory talk is worth further 
scrutiny particularly in view of the results from the Raven’s Matrices tests and our evaluation of the 
curriculum outlined in Section 5. 


We recommend that: 


e School 21 review the extent to which students engage in exploratory talk across subjects 
and how this is promoted across the school. 
e School 21 develop further 'oracy in every lesson’ documentation and resources that focus 


more strongly on exploratory talk. 
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7. Findings: Whole school oracy culture 


7.1 Introduction 


Establishing a whole school oracy culture is an integral part of School 21's approach to developing 
students’ oracy and supporting their learning across subjects. We begin this section by reviewing the 
approaches taken by School 21 to build a whole school oracy culture. School 21's position as a new 
school with a small school population® has enabled it to place oracy centrally within all school activity. 
We therefore also consider the issues other schools may face in trying to establish an oracy culture. 


7.2 Building a whole school oracy culture in School 21 


School 21 plans a variety of opportunities for oracy which help to generate what the staff team view 
as a ‘culture of talk’ at the school. These include student-led parents’ evenings during which students 
talk their parents through their achievements, and peer-to-peer and tutor coaching, e.g. Year 8 pupils 
coach Year 7 students to develop their Ignite talks. This multi-stranded approach has raised the 
status of oracy and generated a shared commitment to oracy development by the teaching staff. As 
the project manager (Visit 2) noted: 


There is no question that we have produced a strong oracy culture within the school which for 
the purposes of this project is both good and challenging and the reason that the teachers are 
so immersed in it. 


Year Group assemblies also play an important role in this regard. One teacher noted that: 


We all do assemblies ... which isn't just talking from the front of but addressing the circles — 
so maybe there is something cultural that is encouraging talk and so on — something cultural 
that is in the air, that is hard to pin down. (Teacher focus group Visit 2) 


As the following field notes from an observed assembly illustrate, assemblies are characterised by 
opportunities for student participation through group discussion: 


Students enter the hall and stand in one large circle. All teachers are present, ready to take 
up roles as group coaches. Each has a notebook and felt pen or mini whiteboard. After 
ringing a bell to signal the start of assembly, the teacher tells the students they are now in the 
court of King Arthur and must use a counting system to decide who will be the next king. He 
asks the students to talk in pairs to predict where someone should sit in the circle if they wish 
to be chosen. He gives them prompts to help them participate in the discussion: 


e What is the question? 
e =| bet you're thinking... 
e /am dying to tell you... 
e What! don't get... 


The teacher rings a bell and the discussions stop. He takes feedback and then asks them to 
move into their coaching groups and share ways of solving the problem. As they discuss, teacher 
coaches listen, sometimes intervening to encourage the students to think about how to manage 
the discussion. After this, they are invited to share solutions. Individuals stand to do so, 
sometimes walking around the room as they explain their ideas. All are invited to share their ideas 
later on a ‘maths challenge board’. (Observation Visit 1A) 


5 School 21 is a new school, which during the pilot period had only Year 7 and 8 students. The Year groups are small, for 
example there were 75 pupils in Year 7. 
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Assemblies are used to explore a range of issues and topics, but are also used to address focus 
areas within the oracy curriculum, for example, staff told us that some assembly time had been 
devoted to exploring vocabulary use in different situations. 


7.3 Establishing an oracy culture in other schools 


School 21 believe that the deployment of the Oracy Curriculum, Culture and Assessment Toolkit ‘can 
only be successful if schools are wholly committed to its implementation and requires Head Teachers 
to be fully engaged in transforming whole school culture’ (Fidoe, 2014: 27). The development of oracy 
is closely related to other aspects of the school’s curriculum and pedagogy. Other emphases in 
School 21 on wellbeing, leadership and enquiry, for example, are seen as both reinforcing and 
benefitting from the oracy curriculum. As the head teacher commented: 


The way you set up a school day flows from that and assemblies and the way you set up a 
classroom and the way you train teachers and their pedagogy so just as you wouldn't line 
them up in rows in an assembly you wouldn't line them up in rows in a classroom. (Head 
teacher interview Visit 2) 


He continued to explain that a key challenge: 


...1S to buy into how you make a school a talking school. ... [oracy] can’t be a sideline or a 
discrete curriculum where you go into an oracy lesson and then the rest of the day you're told 
to line up in lines and told to be silent, there's just no logic to that so a school has got to 
embrace the back-cultural piece more importantly than the other elements because otherwise 
it doesn't work. (Headteacher interview Visit 2) 


This raises questions about the extent to which it is possible to translate the Oracy Curriculum, 
Culture and Assessment Toolkit to other settings. It is also acknowledged that School 21's size and 
whole school oracy culture make positive engagement far more likely. As one teacher explained: 


Every single teacher is aware of the four different strands — the physical, social/emotional, 
cognitive, linguistic...and that the cognitive and the linguistic are really the two areas for 
development...everyone is aware and talks to their particular coaching groups about those 
elements. It's easier to do if you are establishing a new school and harder to do if you are 
fighting your corner in a larger school that has a vision already established...traditionally | 
think oracy would be seen as English and drama departments and possibly not as relevant for 
other people. (Telephone interview, oracy teacher) 


This teacher also acknowledged that a focus on oracy is not necessarily new for schools. As one 
teacher commented, an oracy curriculum is not: 


...brand new altogether, it's often just a formalising of good practice that's probably already 
going on in a lot of good schools but | guess trying to get people to be more analytical with 
what they are doing...In schools I've taught in before you've been in the habit of saying he's 
good at public speaking, she’s not as good and you're not looking at analysing the different 
areas. 


7.4 Summary and recommendations 


School 21 plans a variety of opportunities for oracy which help to generate a ‘culture of talk’. The 
School has reworked traditional school practices, such as assemblies, staff development sessions, 
and parents’ evenings, to ensure that talk plays a central role. This multi-stranded approach has 
ensured that oracy has high status within school and there is a shared commitment to oracy 
development by the teaching staff. Students know that talk matters and that what they say will be 
listened to. It is also worth noting the reciprocal relationship between the oracy curriculum and other 
provision at School 21 where the development of oracy is closely related to other aspects of the 
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school’s curriculum and pedagogy. Emphases on wellbeing, leadership and enquiry, for example, 
appear to both reinforce and benefit from the oracy curriculum, and the school benefits from specialist 
drama teaching. 


School 21’s position as a new school with a small population may also mean that positive 
engagement is more likely as staff and students join a school where oracy has been emphasised from 
the start. The school also benefits from the expertise of a very experienced specialist drama teacher. 
These school-specific factors raise questions about the extent to which it is possible to translate the 
School 21 approach to other settings. It is our professional opinion that the school are right to draw 
the boundaries of their proposed intervention broadly to include oracy across the curriculum and 
whole school culture as well as the Oracy Skills Framework and the dedicated Year 7 curriculum. 
However, it may be that its success is dependent on multiple aspects of the very specific conditions at 
School 21. 


We recommend that: 


Any further evaluation of a scaling up of the intervention would need to consider how School 21's 
oracy approach is interpreted in other schools, and on the opportunities and barriers that arise when 
implementing these approaches in other sites. 
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8. Findings: Oracy Assessment Toolkit 


8.1 Introduction 


In this section we evaluate the Oracy Assessment Toolkit. As described in Section 2.1, the Oracy 
Assessment Toolkit, developed by the University of Cambridge, comprises: a set of three oracy 
assessment tasks that can be used to diagnose and track students’ progress in oracy — a formal 
presentational speech, an instructional activity and a group discussion; a set of six additional tasks to 
be used for assessment for learning; and a rating scheme for assessing students’ performance on 
these tasks and for giving students feedback. The rating scheme enables teachers to give students a 
rating for each of the skills from the Oracy Skills Framework relevant to each task. 


The evaluation questions set out in the protocol focus on assessing the robustness of the Oracy 
Assessment Toolkit as an independent measure of oracy that can be used for testing 
purposes by considering consistency of administration, reliability and validity. We start this 
section by addressing reporting on these issues. 


It is however very important to note that the University of Cambridge team clearly state that their 
Oracy Assessment Toolkit has been designed primarily to provide a useful and usable tool for 
teachers who can use the assessment tasks to draw skill profiles of individual students and plan 
relevant teaching on the use of spoken language. As the team make clear: 


As we are not scoring children in a way that will involve comparing their performance across 
schools or even within schools, reliability is not a major concern. What is important is that the 
ratings given form a useful and informative basis for teachers to give feedback to the students 
and help them move their learning forward. (Mercer et al., 2014: 31) 


In recognition of this key aim of the Oracy Assessment Toolkit we also consider in this section the 
extent to which the Toolkit provides a usable and useful tool for teachers. 


8.2 Consistency of administration of the assessment tasks 


The evidence from the University of Cambridge report (Mercer et al., 2014) suggests that the Oracy 
Assessment Toolkit was being administered consistently. Our own examination of the videos of 
students undertaking the various oracy assessment tasks suggested that the task protocols were 
being adhered to and the students were provided with consistent instructions on each occasion they 
were assessed. It should be noted that the original instructional task — the Map task — was deemed to 
be inappropriate for a number of reasons particularly relating to the students' understanding of what 
they were required to do and what their partners could see on their own maps. In light of this the 
University of Cambridge team replaced this task with the Lego task at the follow-up test stage which 
was a significant improvement from students’, teachers’ and the experts’ perspectives. 


8.3 Rating scheme 


The University of Cambridge set out using a three way rating scheme — gold, silver and bronze. This 
was based on a mastery model where students demonstrate each skill either consistently, or only 
some of the time, or not at all. In the Oracy Assessment Toolkit teachers are asked to rate students 
as GOLD if they ‘consistently demonstrate this skill’, as SILVER if they ‘demonstrate this skill some of 
the time’ and BRONZE if they ‘rarely or never demonstrate this skill’. Our evaluation found mixed 
responses to the three way rating scheme. For example, a teacher we interviewed at School 21 was 
broadly positive about the three point scale: 


Actually the bronze, silver and gold, there's something there that works...I'd initially thought 
that I'd want more strands but | think the three actually worked quite well and you do find that 
somebody broadly comes into one category. 
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The Cambridge team found that many teachers wanted to be able to discriminate between 
performances in a more fine-grained way, and as a result the final rating scheme comprises 7 points 
by allowing the use of + and - within the scale. The seven point scale has the potential to lead to 
better discrimination between students and within students across time. However, the finer grained 
scale may result in teachers adopting a more instrumental approach to the assessment rather than 
using the assessment as a way of thinking more deeply about their students' needs, strengths and 
progress. 


The positive language used in the rating scale was welcomed. As a School 21 teacher observed: 


When I've used that language with children they've liked that idea of being gold, silver or 
bronze...| like the idea that it's quite positive, it's a medal whatever you are getting and you 
can say to them you might be bronze overall but you have this element of silver here. 
(Teacher interview, Visit 2C) 


8.4 Reliability of the Oracy Assessment Toolkit 


The reliability of the Oracy Assessment Toolkit was assessed independently at Sheffield Hallam 
University through a teacher workshop. Five English teachers from local schools attended the 
workshop and used the assessment tasks to rate students’ oracy skills. The teachers rated videos of 
students from School 21 undertaking the Lego, Talking Points and Presentation tasks. The video 
recordings were from the follow-up assessments of students undertaken in April or May towards the 
end of their first year of the oracy intervention. The teachers were presented with five video clips; two 
each of the Lego and Presentation tasks and one of the Talking Points task. For each task the 
teachers were given an overview of the task and the instructions produced by the University of 
Cambridge team and also used in their testing of the Oracy Assessment Toolkit. The teachers were 
also given an explanation as to how to grade each skill being assessed in terms of the three point 
Gold, Silver and Bronze rating scale. The teachers then viewed the videos and gave their ratings on 
the ratings sheets provided to them. Upon completion of all the ratings tasks there was a 15 minute 
discussion of the teachers’ experience of using the Oracy Assessment Toolkit to assess the students’ 
oracy skills. 


Reliability was assessed in a number of ways. First the five teachers were compared to see how often 
they agreed ratings for each of the skills for each of the tasks. In order to achieve this each possible 
pairing of the teachers was analysed to calculate agreement on each assessed skill. With five 
teachers this meant that there were 10 possible pairings to establish levels of agreement (teachers 1 
&2,1&3,184,185;28&3,284,2&5,38&4,3&5, 4&5). The number of times the teachers 
agreed for each skill associated with each task is presented in Table 8.1. This table demonstrates at 
least moderate levels of variability in the number of times judges agreed with each other on the 
various ratings tasks. Given that there are three possible grades given by the judges we would expect 
approximately 1 in 3 ratings to be in agreement by chance. As can be seen in Table 8.1 most ratings 
were numerically above chance. For all of the ratings given with the exception of overall ratings for the 
Lego and Talking Points tasks there were 10 pairwise comparisons (one rater did not complete the 
overall ratings for the Lego and Talking points tasks and so for these there were six pairwise 
comparisons). Chance level agreement would therefore be 3.3 for all but the overall ratings for the 
Lego and Talking Points tasks (where chance would have been 2). For the Lego task it is apparent 
that the level of agreement between the raters is better for video 1 than it is for video 2. This is a 
concern as with a reliable assessment tool we would expect good levels of agreement for both of the 
videos. There is perhaps a greater level of consistency for the Talking Points task but even here there 
are a number of times where the raters have failed to agree above chance levels. Particularly 
concerning here is the poor performance on the items 'Is willing to listen' where there is below chance 
agreement for all three students being rated. This item should perhaps be modified to make it more 
reliable. It is interesting to note that in the University of Cambridge report (Mercer et al., 2014) there is 
a suggestion that this item be amended to reflect active rather than passive listening. For the 
Presentation task there is good agreement for the first student but relatively poor agreement on many 
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of the items for the second student. Again this is worrying as we would expect above chance 
agreement for all students being rated. 


Along with this descriptive analysis inter-rater correlations were calculated as an alternative measure 
of inter-rater reliability. Following the procedure outlined by Hatch and Lazaraton (1991), Pearson 
Product-Moment correlations coefficients were calculated for all judges and then to avoid issues of 
skewness a Fisher z transformation was applied to these. These transformed correlations were then 
entered in to the following formula: 


NYap 
aa + (n—1)razg 

where ry is the overall reliability of the judges, n is the number of raters and rag is average correlation 
between the raters (this latter was the z-transformed average correlation). For each student rated on 
the Lego task the inter-rater reliabilities were comparatively low (0.51, 0.44, 0.65 and 0.48). For the 
Talking Points task the reliabilities were better but still a little low at 0.63, 0.67 and 0.66. Finally, for 
the presentations the reliabilities were 0.52 and 0.71. Overall, these reliability coefficients would lead 
us to question the reliability of the assessment tool. However, it has to be remembered that the 
teachers undertaking the ratings in this analysis were not trained in using the oracy tool and as such 
we would expect greater variability in the ratings given than for raters who have been trained. Hatch 
and Lazaraton (1991) suggest relatively low reliability coefficients may be the result of untrained 
raters and this may be the case in the current analysis. 


Table 8.1: Number of times raters were in agreement for each of the oracy tasks 
Task AV Te [=oym § Vite [-foyyd 


StudentA StudentB StudentA StudentB 


3 i) Uses vocabulary to suit topic, purpose and situation 6 4 1 3 
4 ii) Register 6 6 3 3 
5 i) Chooses and organises content to convey meaning 4 4 4 3 
and intention 


6 i) Seeks information and clarification through 2 10 6 4 
questions 


7 i) Maintains focus on task 10 10 6 6 
9 i) Takes account of level of understanding of audience 4 10 2 6 
11 iii) Takes turns appropriately 6 4 3 4 
12 ii) Responds appropriately to questioning 6 6 4 4 
13 iii) Participates actively in discussion 4 6 4 4 
Overall assessment* 1 3 3 0 
Talking Points Task: StudentA StudentB Student C 

2 iii) Makes eye contact 6 2 5 

6 i) Seeks information and clarification through 6 6 2 

questions 

7 i) Maintains focus on task 6 6 4 

8 i) Gives reasons to support views 10 6 6 

11 ii) Sustains dialogue 6 4 8 

11 iii) Takes turns 4 4 3 

12 i) Responds to and builds on views of others 3 4 2 

12 ii) Responds appropriately to questioning 6 6 6 

12 iv) Is willing to listen 1 2 2 

13 iii) Active and whole-hearted participation in 3 4 6 

discussion 

Overall assessment* 3 3 1 


Presentation Task: StudentA Student B 
1 i) Clarity and projection of voice 6 6 
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1 ii) Tonal variation 6 3 
2 i) Gesture 4 4 
2 ii) Posture 4 6 
6 Use of metaphor, humour, irony, mimicry and other 6 4 
rhetorical devices 


3 Vocabulary choice to suit topic, purpose and situation 6 4 
5 Fluency and flow of talk 6 1 
7 i) Choice and organisation of content to convey 6 2 
meaning and intention 

9 ii) Time management 6 2 
11) Taking account of level of understanding of the 6 1 
audience 

14 i) Self-confidence 10 2 
14 ii) Liveliness, flair 4 2 
Overall assessment 10 1 


* These rating totals were based upon 4 raters rather than 5. 


One of the reasons why the assessment of oracy is commonly understood to be problematic is 
because it is often seen as subjective. It is not surprising therefore that Sheffield focus group teachers 
commented on the subjective nature of assessment based on these tasks and on the different ways in 
which the criteria could be interpreted. For example, one Sheffield focus group teacher noted her 
personal preferences in how people presented ideas to her: 


| quite like people that maybe seem a little bit unsure of what it is that they’re saying, and | 
think that that’s maybe a deliberate presentational ploy that a student might use. You could 
have argued that the girl was deliberately hesitant as she’s talking to Year 6s and she’s 
thinking carefully about what they might need to know and what they don’t. There’s that 
whole, kind of, boys get assessed as better speakers sometimes because of this narrow view 
of confidence as well, so that where it got to self-confidence and liveliness and flair at the 
bottom, that was where | felt there was most room for who are you talking to, and about what, 
and in what situation? 


While assessment of oracy is always likely to be subjective to some degree, the University of 
Cambridge team reported a reasonable level of inter-rater reliability in their trial of the Oracy 
Assessment Toolkit. They report inter-rater reliability at the level of 0.77 (Mercer et al., 2014). This 
level of reliability was reported for the post-intervention presentation task and so given our relatively 
low reliability ratings for the Lego task it would be good for the University of Cambridge team to report 
specific reliability statistics for this task in particular. 


The University of Cambridge team emphasise the need to ensure teachers have opportunities to 
moderate judgements and talk together, as they explained in a telephone interview: 


[Our] intended way around this issue [reliability] is by our bank of video exemplars because 
the easiest way to learn how to rate these kinds of tasks, | mean from the videos, is through 
seeing examples that benchmark the standards so that teachers can look at what their 
children have done and look at what's in our bank and say ah right that's what a silver 
presentation looks like and this is why... and that to me seems an ideal way to address 
validity concerns for this kind of scale. 


Planning for such opportunities would seem to be highly appropriate particularly as this approach is 
very much in line with existing moderation procedures including those used extensively for the 
assessment of writing. The University of Cambridge team’s inter-rater reliability analysis was based 
on assessments made with better trained raters and as noted above they reported generally higher 
correlations than we report here. On the basis of the evidence from our test of the Oracy Assessment 
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Toolkit and the testing reporting by the University of Cambridge we would suggest that the Oracy 
Assessment Toolkit is only reliable if it is utilised by appropriately trained raters. 


8.5 Validity of the Oracy Assessment Toolkit 


The validity of the Oracy Assessment Toolkit can be evaluated in a number of ways and as with the 
validity of any assessment tool the evaluation of validity is an ongoing process. As we have 
highlighted the oracy assessment tasks were designed to align with the Oracy Skills Framework 
developed by Mercer et al. (2014). Indeed the individual items which are rated for each of the tasks 
have been directly developed from this skills framework. As such it can be argued that the Oracy 
Assessment Toolkit has a good deal of validity. The Oracy Skills Framework highlights aspects of 
oracy that have not been considered in previous attempts to develop assessments of oracy. The 
Sheffield focus group focused particularly on the physical strand, for example, one teacher 
commented: 


Where we’ve got clarity and projection of voice, and tonal variation, gesture and posture — 
tonal variation | found quite helpful, because | feel like that’s something that doesn’t currently 
exist in criteria that | have to apply within English, and yet it’s something that you really do 
want to be able to see in the way that a kid is expressing themselves. 


| can see a place for posture, like, for example, with that first child. He was constantly 
fidgeting, and, you know, it is important that you focus your audience by standing in a certain 
way, you know, commanding the room, that is a part of presentation. 


Validity can also be measured in terms of how well experts and practitioners accept it as a valid 
instrument (this is often called content validity). The University of Cambridge conducted video review 
days with teachers who had used the oracy assessment, teachers who had not been involved in using 
the toolkit and a panel of oracy experts. The data from the Cambridge team's evaluation of the Oracy 
Assessment Toolkit suggest that there is variability in validity in terms of the particular tasks. In this 
regard it would appear that the Lego task has a high level of validity. The experts and teachers from 
the Cambridge trials were particularly positive about this task in enabling students to demonstrate 
their ability to engage in exploratory talk. The Talking Points task however appeared to attract fewer 
positive comments as it was apparent that in many cases the students were not discussing the topics 
as such but quite mechanistically listening to each other's views. As a teacher we interviewed at 
School 21 explained: 


The children at this school are very used to the idea of taking turns and they are very good at 
that but they didn't actually engage in any real debate. So what tended to happen was that 
each child said their point and they sort of just took turns very statically. (Teacher interview: 
Visit 2C) 


When observation and students and teacher feedback indicated that tasks were not engaging in 
purposeful talk they were revised, as one of the University of Cambridge team explained: 


We did modify the Talking Points task from the initial task to the end task because when we 
looked at some of the items on Talking Points we felt that some of them didn't problematise 
issues enough or didn't create a dilemma ... the one that people responded to most was that 
footballers are paid too much. There was a real genuine debate and clash of opinions and we 
looked at other items and we thought actually they are a bit anodyne. In the end we reduced 
the number of tasks and made them more problematic in nature. 


Tasks had also been designed to be stand-alone to provide a ‘level playing field’ for all students, and 
teachers in the Sheffield focus group agreed that this had been achieved: 


What | liked about that as well was it made an automatic level playing field to some extent. | 
felt like in the group discussion, the lad on the left, you know, | agreed with some of his views, 
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but he had clearly had discussions before and thought about these ideas before ... whereas | 
felt like sometimes there’s a cultural advantage in a discussion, isn’t there? Like in terms of 
your richness and your knowledge and the experiences that you’ve had, whereas that Lego 
task, pretty much everybody can have a go and demonstrate a real range within that. 


Regarding validity, it is worth noting that teachers in the Sheffield focus group felt that the three way 
rating scheme led them to judging the frequency with which students used certain skills rather than 
how they did so. One of the Sheffield focus group teachers felt that this approach prompted her to 
focus on frequency not quality: 


| feel like on that task | ended up doing, when it came to the overall assessment, doing this, 
kind of, accrued sense of how many S’s and G's and B’s have | put up, and then counting it 


up. 


Another Sheffield focus group teacher, who also found herself focusing on frequency, commented 
that frequency in relation to some skills could be detrimental: 


Gesture, you could imagine somebody sitting there going, ‘Yes, that boy made loads of 
gestures’ — loads of fruitless gestures, repetitive meaningless gestures, whereas the girl was 
much more strategic in her use of gesture. That, when you compare gesture to, you know, 
consistently demonstrate this skill, like you could accidentally over-reward that boy. | don’t 
know, it seemed a bit of a clumsy thing. 


The decision to ask teachers to rate in terms of varying levels of consistency was taken by the 
University of Cambridge team to make it easier for teachers to arrive at appropriate judgements, and 
they reported that this had worked effectively for the teachers they worked with, who had received 
some training in use of the tasks. The Cambridge team felt that this training, or further guidance, is 
necessary to explore what is meant by ‘consistency’: 


You're always working with that judgement of quality as well ... that's the kind of thing we try 
and explain and it's a case of trying to explain that down to some guidelines that take both of 
those aspects into account. 


Relating to the 3-point ratings scheme some of the teachers at the independent Sheffield Hallam 
University focus group would have preferred a more finely graded scale: 


| would have liked a finer grading within it, | think, because | found it difficult. Like [another 
teacher in the group] said, | was putting silver for a lot of them, but didn’t feel that they were 
on a similar level in terms of their oracy. 


It is interesting to note however that the 3-point ordinal scale we used at our testing session has now 
been replaced by the University Cambridge team with a 7-point ordinal scale. It is as yet unclear if this 
will address the concerns outlined above about measuring frequency rather than the quality and 
consistency in using that skill (see Section 8.3). 


The Sheffield focus group teachers also expressed concerns about the use of other terms that could 
have been used with regard to the rating process. They briefly agreed that they needed to consider 
‘appropriate’ use of talk, but went on to discuss that ‘appropriateness’ too is problematic: 


| agree with [another focus group teacher] in terms of the use of that word ‘appropriate’. It’s 
always a bit of a problem when you find that in criteria. It became a bit meaningless in such a 
specific set of circumstances. 


A further concern expressed during the Sheffield focus group was that it might be possible to ‘teach to 
the test’, and therefore undermine the validity of the test, as has happened in other areas of language 
and literacy education, leading to a narrow and atomised provision for oracy: 
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The other issue is how easy would it be to, kind of, train a child to be able to fill these criteria 
without them actually being a good speaker. | know that’s true in lots of areas of English, but 
that was something | found. | felt uncomfortable especially with the presentation one with that. 
| felt it could fit and it didn’t actually mean that they were a good speaker. 


The Oracy Assessment Toolkit appears to have a good level of face validity with the students that the 
Cambridge team interviewed indicating that they thought the tasks were appropriate for measuring 
their oracy skills. 


In summary we consider that the Oracy Assesment Toolkit provides a reasonable level of validity and 
that the evidence of validity we were able to examine supports the claims for validity made by the 
Cambridge team. 


8.6 Training and support for using the assessment tasks and rating criteria 


While there is not as yet a test handbook, the Cambridge team has produced a number of supporting 
resources which have been made available via a dedicated website 
(www.educ.cam.ac.uk/research/projects/oracytoolkit/). The landing page for the Oracy Assessment 
Toolkit provides an overview of the rationale for development of the toolkit as well as a video of 
Professor Neil Mercer introducing the Oracy Assessment Toolkit and the Oracy Skills Framework. 
From this landing page there are links to the skills framework and the toolkit itself. On the Oracy Skills 
Framework page the image does not load properly but there is a link to a PDF file which has the 
image detailing the four strands of the framework. There is also a link to a document which contains a 
glossary of the skills highlighted in the Oracy Skills Framework. This is particularly useful as it 
provides guidance on what would be expected of an appropriately skilled speaker in relation to the 
items contained in the framework. It would have been useful in this section of the site to provide the 
rationale for the Oracy Skills Framework and a summary of how it was developed. This would aid the 
understanding of the function and utility of the Oracy Assessment Toolkit. 


The next section of the site provides an overview of the Oracy Assessment Toolkit tasks. It provides a 
brief description of the oracy tasks along with an example assessment form. This provides a helpful 
overview of the oracy tasks themselves and there are additional links to the specific tasks with links to 
task instructions. 


The guidance on completing the assessment sheets on the website states that 'Teachers have 
adopted a variety of approaches to completing these sheets and there is no ‘correct way’. If the tasks 
are to be used for professional development purposes and as a basis for professional discussion this 
appears appropriate. However, it may undermine the drive for consistency in the use of the toolkit and 
the assessments of oracy, particularly by teachers who may be relying solely on the web materials for 
training. While it is recognised that the tasks are intended to be ‘teacher friendly’ perhaps there needs 
to be clearer guidance on how to complete the sheets to ensure reliability and consistency. 


This section of the website also provides teachers with both the simple 3-point Gold, Silver and 
Bronze ratings scale and a finer-grained 7-point scale which enabled raters to give grades with '+' and 
‘-" ratings. Since the University has now adopted a 7-point scale the instruction sheets and proformas 
will require amendment. 


The specific task pages would be enhanced by providing more detailed descriptions of the tasks, 
along with a rationale for their development as part of the Oracy Curriculum, Culture and Assessment 
Toolkit. In addition, the links to the PDF files could have more meaningful descriptions rather than just 
consisting of the filenames. This would make it clearer to teachers and researchers what the links are 
pointing to. The link to the Lego task has images of the models which students are trying to build. If 
these are the images that the students are describing to their partners this needs to be made explicit. 
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The documents in the Assessment for Learning tasks section site provide very good introductions to 
the tasks and a clear rationale for each task. It would be helpful for this level of detail to be provided in 
the explanations of the main oracy assessment tasks (i.e. the individual presentation, the instructional 
Lego task and the group Talking Points tasks). 


There is a link provided to a Record Sheet but this does not provide sufficient information on the 
purpose of the sheet and or how it can be effectively utilised by teachers. 


The final link is to the web repository which hosts the exemplar videos of students undertaking the 
oracy assessment tasks. This webpage provides access to nineteen videos from across the spectrum 
of oracy skills for each of the assessment tasks. Along with each video there is a very useful 
description of why that video has been graded in the way that it has. This is illustrated by this 
description from the Talking Points Silver Minus exemplar: 


Look at the girl in the middle, who presents a Silver minus performance. She interacts well, 
making clear eye contact with both of the other participants; she listens and responds actively 
throughout the discussion. She is willing to take turns, though the turn-taking in this group is 
not systematic. 


The videos and commentaries provide an excellent resource for teachers to develop understanding of 
oracy and are the most effective means of ensuring that the Oracy Assessment Toolkit is used 
consistently by teachers and researchers and thus of ensuring higher levels of reliability. We therefore 
suggest that more needs to be made of the importance of videos and commentaries for training 
purposes and to support teachers’ discussion of oracy in the introduction to the Oracy Assessment 
Toolkit on the University of Cambridge website and on the Voice 21 website. The initial page with the 
link to the external site requires information relating to the purpose of the exemplar videos and indeed 
should be reinforcing the importance of engaging with these as part of teachers’ training. 


8.7 Use of the Oracy Assessment Toolkit within the School 21 curriculum 


During the period of this evaluation the initial and follow-up tests were conducted by oracy teachers at 
School 21 and at least one of the oracy teachers made some use of the assessment for learning 
tasks. In addition School 21 have developed and begun to use an oracy Assessing Pupil Progress 
(APP) grid to track student progress against the skills in the Oracy Skills Framework. At this stage it is 
not clear how the Cambridge Oracy Assessment Toolkit is intended to fit within the overall Oracy 
Curriculum, Culture and Assessment Toolkit. It would be helpful for this to be articulated. There is also 
a need to consider the relationship between the Voice 21 website and the University of Cambridge 
website and the explanations that are placed on both to explain this relationship. 


8.8 Summary and recommendations 


In our opinion the Oracy Assessment Toolkit could potentially provide valid and reliable measures of 
students’ oracy skills, however more data are required before this can be firmly established. There is 
good evidence from the University of Cambridge team that they take the reliability and validity of the 
Oracy Assessment Toolkit very seriously. They have changed one of the assessment activities 
already (from the Map Task in the original design to the Lego Task currently) and this has led to a 
significant improvement in the toolkit. Additionally, they have listened to the feedback from their own 
experts and raters concerning refining the grading scheme. The original scheme allowed raters to use 
just Bronze, Silver and Gold ratings to judge each skill associated with each of the oracy tasks. 
However, feedback indicated that a finer-grained scale was needed and so the scale was adjusted to 
incorporate -/+ ratings. This increased the scale from a 3-point scale to a 7-point scale (Bronze, 
Bronze+, Silver-, Silver, Silver+, Gold- and Gold). The team also acknowledge that appropriate 
training is vital to the reliability of the Oracy Assessment Toolkit. Thus is it recommended that all 
documentation provides clear advice that the tool should only be utilised by appropriately trained 
raters. To this end the Cambridge team have made available exemplar videos which provide 
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prospective users with examples of students with differing abilities performing the oracy assessment 
tasks, together with a brief narrative of what skills the students are demonstrating. It is also important 
to recognise the potential value of these exemplar rating videos and commentaries in supporting 
teacher talk about oracy and teacher professional development. This potential use is not yet 
sufficiently visible on either the University of Cambridge or the Voice 21 website. 


A tension exists between developing an assessment tool with very high levels of reliability that could 
be used in a large scale trial and producing an assessment tool that is usable by, and useful to, 
teachers. The University of Cambridge have been clear from the outset that their intention is the latter. 


We recommend that: 


Further reliability analyses are undertaken particularly on the use of the 7-point ratings scheme as 
well as a proper item-analysis. In order to do this a larger sample of raters would be required. 


The University of Cambridge continue to review the assessment tasks to ensure validity as they are 
used in different contexts. 


Exemplar videos to support teachers in undertaking reliable assessments and as a basis for teacher 
professional development are produced. Videos exemplifying achievement at Gold level for the talking 
points task would be particularly useful. 


The way in which the Oracy Assessment Toolkit is presented on the University of Cambridge Oracy 
Assessment Toolkit website is reviewed ensuring that: 


e some context to, and rationale for, the Oracy Skills Framework is provided 

e clearer and more detailed descriptions of the contents of each of the task-specific pages are 
presented along with more meaningful named links to documents 

e clear advice on using the assessment sheets to enhance reliability is provided. 


The way in which the Oracy Assessment Toolkit is intended to be used within the overall Oracy Curriculum, 


Culture and Assessment Toolkit intervention is clearly articulated and a clearer explanatory link is made between 
the University of Cambridge website and the Voice 21 website. 
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9. Supporting CPD and resources 


9.1 Introduction 


Given the developmental nature of the project we are unable to provide an assessment of the CPD 
that would be available to schools adopting the Oracy Curriculum, Culture and Assessment Toolkit. In 
this section we therefore report briefly on School 21's teachers’ perceptions of the effectiveness of the 
oracy CPD that took place during the development period and then focus on the proposal set out in 
School 21's final project report (Fidoe, 2014) for CPD for other schools. We also present a 
professional review of Voice 21, the website developed in this project (http://voice21.org/). The 
materials on the University of Cambridge website that support the use of the Oracy Assessment 
Toolkit were reviewed in Section 8. 


Stoll et al.'s (2012) literature review identified nine strong claims about effective professional 
development, namely that effective professional development: 


starts with the end in mind (i.e what are the intended outcomes? particularly for students) 
challenges thinking as part of changing practice 

is based on assessment of individual and school needs 

involves connecting work-based learning and external expertise 

comprises professional learning opportunities that are varied, rich and sustainable 

uses action research and enquiry as key tools 

is strongly enhanced through collaborative learning and joint practice development 

is enhanced by creating professional learning communities within and between schools 
requires leadership to create the necessary conditions. 


OONOAPWNE 


We draw on these criteria to inform our consideration of the CPD experienced by teachers at School 
21 during the pilot and the effectiveness of School 21's proposals for CPD to support the roll-out of 
the Oracy Curriculum, Culture and Assessment Toolkit. 


9.2 Oracy CPD at School 21 during the pilot 


School 21 has from its start-up, embedded a whole school oracy culture which integrates the 
development of students’ oracy skills with other school aims such as student wellbeing and 
leadership. It is therefore somewhat difficult to draw a clear boundary around CPD that is specifically 
designed to support the development of oracy from those CPD activities that have the broader aim of 
supporting the development of School 21's wider curriculum and culture. The head teacher and 
project manager pointed to whole school oracy focused CPD events, coaching, assemblies and the 
visuals in all classrooms as all contributing to supporting all teachers in developing their oracy 
teaching and/or the use of oracy to support learning in their subject. 


The project manager (Interview, Visit 2) explained: 


We've not measured impact on teachers yet, however there is a huge impact on all the 
teachers and you see it in the practice of all the lessons. 


Although the impact of oracy CPD has not yet been measured it was evident from all interviewees 
that there is a clear focus on intended student outcomes, or as Stoll et al. (2102) phrase it, CPD 
‘starts with the end in mind’. The Oracy Skills Framework has the potential to be particularly helpful in 
supporting teachers' CPD and providing a common language to talk about oracy. The end of the 
project report (Fidoe, 2014) matches some oracy techniques that teachers may use in any subject 
with skills identified in the Oracy Skills Framework. Further development of this approach is likely to 
enhance support for teachers. 
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When teachers were asked about the CPD they had received around oracy they all pointed to 
assemblies, identifying them as important in helping them learn oracy approaches. All staff are 
present in assemblies and students work in groups of 12 within their form. The teachers leading the 
assembly model approaches that other teachers can use within their own teaching. As one teacher 
explained: 


So then certainly things like the pairs, how children work together in pairs, how children work 
together in threes, the rest of the staff would understand that and would also understand the 
language of oracy that we use like the instigator, the builder, the competitor, every member of 
staff has got that up on their wall. (Teacher interview, Visit 2C) 


This process of interaction and mutual development between teachers aligns with the notion of joint 
practice development which Fielding et al. (2005) demonstrated supports the exchange of knowledge 
between teachers and is recognised as contributing positively to effective professional development 
(Stoll et al., 2012). 


Teachers emphasised the importance of the scaffolding provided by the common protocols for oracy 
developed in this project and the oracy visuals that are placed in every classroom. A teacher focus 
group also noted that as the school grows in size and more teachers are recruited, maintaining 
consistent approaches will become more challenging: 


We have to grow the consistency — how we induct staff and keep the training regular — but not 
over-kill — having that consistency but still having innovation. So by having common protocols 
and using that, we try and keep it consistent. With any sort of new model that you are trying to 
embed you need that. (Teacher focus group, Visit 2) 


In addition oracy teachers participated in training from the University of Cambridge team using the 
prototype assessment tool and attended an assessment moderation event at the university. They 
were also supported by external drama coaches. The use of external expertise to support 
professional development is a key feature of effective professional development (Stoll et al., 2012). 


9.3 Proposed CPD package for other schools 


Developing an appropriate approach to CPD to support other schools to implement the Oracy 
Curriculum, Culture and Assessment Toolkit presents a number of challenges. School 21 had the 
advantage of being a new school, starting in their first year with only Year 7 students and a small staff 
group recruited with the school’s distinctive ethos in mind. They were therefore able to embed an 
oracy culture and high expectations of students’ progress and achievement in oracy from inception. In 
contrast, the CPD required by staff in other schools will need to take account of established cultures 
and practices and, in most cases, impact on a larger staff group that may primarily be engaged in 
more tightly subject-focused professional learning activities and communities. 


In a potential CPD package, the head teacher (Interview, Visit 2) of School 21's aspiration is for a 
tailored coaching approach: 


We don't like training where someone just sort of comes in and delivers something and goes 
again and what we will do in local schools is all time-consuming and we will work alongside 
them and actually find out from them what they need and what they're expecting, there'll be a 
huge variety and therefore we'll have a set of programmes and toolkits that will apply to 
anyone but | want to tailor it to what the individual schools require at whatever stage they're 
at. 


Such an approach has the potential to address the complexities of integrating an oracy curriculum 
and culture into more established schools and there is a substantial body of evidence that supports 
the effectiveness of coaching as a school change/improvement strategy (See for example Simkins et 
al., 2009). This intended approach also incorporates a number of the key features of effective 
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professional development identified by Stoll et al. (2012), namely: it is based on assessment of 
individual and school needs; it involves connecting work-based learning and external expertise; it 
comprises professional learning opportunities that are varied, rich and sustainable; it is strongly 
enhanced through collaborative learning and joint practice development; and it creates professional 
learning communities within and between schools. 


The package of CPD proposed by School 21 is set out in Section 2.1. The engagement of senior 
leaders in CPD, as proposed, would appear to be essential. It is fundamental to helping schools 
develop an oracy culture as well as ensuring that leaders have sufficient understanding of the 
intervention to create the necessary conditions to enable teachers to implement the toolkit — a 
necessary condition of effective professional learning (Stoll et al., 2012). As Section 7 has identified, 
developing a whole school oracy culture is a core feature of the School 21 approach, but one that 
may be difficult for other schools to embed, as a School 21 teacher observed: 


We all do assemblies ... which isn't just talking from the front but addressing the circles — so 
maybe there is something cultural that is encouraging talk and so on ... something cultural 
that is in the air, that is hard to pin down. (Teacher focus group, Visit 2) 


At this stage it is not possible to ascertain whether the proposed CPD package will be sufficient, 
particularly in terms of the amount of coaching provided, to support schools to embed oracy; such a 
judgement would require assessment when the package is piloted. 


9.4 Voice 21 website 


The following evaluative commentary is based on a review of the Voice 21 website: by two members 
of the evaluation team on 24 November 2014; and by members of the Sheffield Hallam University 
Language and Literacy Education Research Group on 24 September 2014. It is recognised that the 
website was still under development when these review meetings were held and that School 21 
believe that the best way to support CPD is face-to-face and that the website might not therefore be 
expected to stand alone. However, as we were not able to observe any CPD activity, we have 
focused attention on the Voice 21 website. 


Our commentary in this section is intended primarily to be used formatively to inform the next stage of 
website development to enable School 21 to use the website as effectively as possible in 
communicating their approach. These comments are intended to be read alongside those related to 
the dedicated Year 7 oracy curriculum and oracy in every lesson as they appear on the website, 
presented in Sections 5 and 6. 


The University of Cambridge website also provides materials to support professional learning for 
teachers using the Oracy Assessment Toolkit. This has been reviewed in Section 8. In this section we 
examine the interrelationship between the Voice 21 and the University of Cambridge websites and 
ways in which that supports and limits their use as CPD resources. 


Structure and organisation of the Voice 21 website 


The structure of the website reflects the elements of the Oracy Curriculum, Culture and Assessment 
Toolkit, i.e. there are links to: the Oracy Skills Framework; the dedicated Year 7 oracy curriculum; 
oracy in every lesson; and building an oracy culture. There are also links to the University of 
Cambridge Oracy Assessment Toolkit. It would be helpful to have some explicit signposting as, when 
we visited the site, our perspective was that key elements, such as the Oracy Skills Framework and 
the curriculum, while present, were not foregrounded. The use of the Oracy Skills Framework in 
guiding curriculum and assessment could also usefully be highlighted. 


The website is attractive and includes a variety of approaches for teachers and leaders from other 
schools to draw upon. It provides a bank of resources which will be useful to teachers and organising 
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opportunities for oracy. The current approach is to let these materials ‘speak for themselves’. Few 
resources include any written commentary, although spoken commentary is given in some videos. 


The inclusion of a large number of videos would seem to be a strength. The videos exemplify 
approaches in an accessible way. It was our view that the most useful videos are those that relate to 
a sequence of videos and are clearly placed within a programme of work — and where this is easily 
apparent to the viewer. 


A more ‘mediated’ approach might provide a more detailed rationale for approaches illustrated by 
videos, along with guidance on how these might be used within a sequence of activities, and 
principles for adapting them for use in other contexts (e.g. linked to other topics or other foci in oracy). 
Drawing on some of the rationales presented in the School 21 final report (Fidoe, 2014) would begin 
to address this issue. Some exemplification of how resources might be adapted for use by students of 
different ages would be helpful. 


Content of resources 


The exemplar resources provided are engaging and practical. It is recognised that School 21 have 
identified resources that might be seen as generic, that could be used and applied in a variety of 
contexts, and it is likely that this approach will be welcomed by teachers. In lessons observed, and in 
lesson outlines provided on the website, there is a clear emphasis on providing meaningful contexts 
for talk (with a range of purposes and audiences). We suggest that it is important that the resources 
on the website are enhanced with the addition of explicit reference to the need to apply these 
approaches within specific meaningful contexts, and for the need to provide diverse contexts for oracy 
and to explore repertoires of talk. 


Suitability for a teacher/school audience 


While it is generally agreed that oracy has not been a priority within the curriculum (e.g. see 
Alexander, 2012), all schools will have addressed speaking and listening and there are many schools 
in which talk has been a focus for development. School 21 believe their curriculum has a distinctive 
contribution to make and our evaluation indicates that the proposed curriculum brings together key 
elements in ways that promise to prove useful to teachers in developing this important area. However, 
while the School 21 approach includes distinctive elements, it would be overstating the case to argue 
that a focus on oracy is either unique or new. Teachers’ existing expertise and prior research 
therefore need to be acknowledged within the materials. Some schools may otherwise not engage 
with the materials if they feel that they offer nothing new, or if they read the materials as discounting 
work developed elsewhere. The need to recognise existing individual and school expertise also links 
to Stoll et al.'s (2012) claim that effective professional development should be based on an 
assessment of individual and school need. 


Identifying the distinctive contribution of Voice 21 


Linked to the previous point, there is a need to clarify what is distinctive about the School 21 approach 
as represented on the Voice 21 website. It would be helpful to more clearly foreground and justify the 
key elements of the Oracy Curriulum, Culture and Assessment Toolkit, i.e. the Oracy Skills 
Framework, the dedicated Year 7 oracy curriculum, oracy in every lesson, an oracy culture and the 
Oracy Assessment Toolkit. These dimensions are present, but the website does not clearly explain 
the link between the overarching Oracy Skills Framework and the specific activities recommended. It 
would also be useful to have further detailed exemplification of how teachers have worked with the 
framework to support pupils’ development, and of the challenges they have faced while doing so. 
Similarly, it is clear that School 21’s provision is informed by the need to provide motivating contexts 
for oracy and yet this is not explicitly stated on the website. At School 21, oracy is carefully 
contextualised within a range of motivating and challenging projects. However, this does not appear 
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to be made explicit. Since other schools may integrate oracy in different ways, it would seem to be 
important to clarify the importance of planning for relevant contexts. 


Need for additional guidance and resources on developing exploratory talk 


Currently, the website provides good exemplification of talk for presentational and formal purposes, 
using talk effectively to argue a point of view, for example, or formally present information or opinion. 
In parts of the website — particularly the rationale for the development of oracy (e.g. see Spreading 
the Word video) — and in some teacher interviews, there is an articulation of the relationship between 
talk and learning. However there were fewer examples of talk for learning on the website. There is 
good support for organising for talk, e.g. through talk protocols, but less for exploring and planning for 
the development of exploratory talk and on its speculative nature. Talk on the Voice 21 website is 
often presented as a means of communicating thinking rather than developing it. Given the 
substantial amount of research which highlights the relationship between talk and learning, this would 
seem to be an area to develop. This might be done, for example, by linking to examples provided 
through other projects or other schools, or by providing examples from School 21 to illustrate how 
these insights from prior research have been applied. 


External research evidence 


The approach to oracy at School 21 has been developed through a process of considerable 
innovation and ongoing review by a dedicated and creative staff team. Given the breadth of what is 
being attempted, and the developmental nature of the project, there is a need to provide clear 
justification for what is being promoted to other schools and also to acknowledge the provisional 
nature of what is being claimed to be successful. In this regard it would be helpful to make explicit the 
sources that have informed the development of the curriculum. For example, there is a link to 
research on the website which lists a number of sources, including research reviews, commentaries 
and two journal articles. It is currently unclear why these sources have been selected or whether (and 
if so how) they have informed the development of the curriculum. 


Using the website to support CPD 


The website is presented as a series of resources, rather than as a structured web-based 
professional development tool. For example, at the time of review the section headed ‘Teacher 
Training’ comprised a series of videos taken from a training day at School 21 but there is no rationale 
for the materials or explication of how teachers visiting the site may draw on them for professional 
development purposes. Although School 21 recommend that the website is used alongside a 
supporting CPD package, the website is publicly available for any school to access. Therefore we 
suggest that consideration is given to how the website might work to support continuing professional 
development of teachers in other schools. This would require, for example, clear guidance on how to 
use the website for professional purposes, taking account of teachers’ prior knowledge and existing 
expertise, activities that encourage teachers to reflect on how the proposed approaches fit with their 
practice, and the needs and strengths of their students. 


Usability of website 


There are some glitches in the Voice 21 website that will need to be addressed as it is further 
developed. For example, some hyperlinks do not work, and there are too many clicks to access key 
documents. 


9.5 Summary and recommendations 


Oracy CPD at School 21 has been positively received by teachers, and both teachers and leaders 
point to assemblies where oracy teaching is modelled, oracy protocols and classroom visuals are 
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important CPD tools. The proposed CPD package to support other schools implementing the Oracy 
Curriculum, Culture and Assessment Toolkit has yet to be piloted. The CPD package design includes 
a number of techniques, such as coaching, that have been shown to be effective in supporting 
teachers’ professional development. 


Since the overall CPD package to support schools in implementing the School 21 oracy approach is 
still to be implemented we focus our recommendations on enhancing the effectiveness of the Voice21 
website as a CPD tool. We recognise that the site is still under development and some of these 
issues may now have been addressed. The website includes sections linked to each component of 
the Oracy Curriculum, Culture and Assessment Toolkit, although the relationships between these 
components could be more effectively signposted. The exemplar resources provided are engaging 
and practical although more could be done to provide a rationale for the School 21 approach and the 
strategies recommended. The website provides good exemplification of talk for presentational and 
formal purposes, using talk effectively to argue a point of view, for example, or formally present 
information or opinion. 


We recommend that: 


The effectiveness of the proposed CPD package should be evaluated when it is rolled out to other 
schools. 


The effectiveness of the Voice21 website could be enhanced by: 


e astructure that foregrounds the Oracy Skills Framework and the other key components of the 
Oracy Curriculum, Culture and Assessment Toolkit and explains the relationship between 
these elements 

e clear rationales and explanations for all materials explaining their purpose and intended use 
in relation to other materials 

e identifying the unique contribution of the School 21 approach as well as acknowledging work 
that is already in place elsewhere and signposting through the site for teachers with differing 
levels of prior knowledge and experience of oracy teaching 

e the addition of materials that exemplify talk for learning. 


As the project progresses it would be helpful to add a section that exemplifies how other schools have 
used the Oracy Curriculum, Culture and Assessment Toolkit. 
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10. Conclusions 


| Key Conclusions 


1. The Oracy Skills Framework provides a useful tool for schools wishing to review and 
develop their approach to oracy. The associated Oracy Assessment Toolkit provides 
teachers with a tool that can be used diagnostically and to track students’ progress in 
developing oracy skills. 


2. The Oracy Curriculum, Culture and Assessment Toolkit as implemented in School 21 
appears to provide a sound foundation for the development of oracy skills, with particular 
strengths in supporting persuasive talk and talk for presentational purposes and in formal 
contexts. This multi-stranded approach may demand some fundamental shifts in approach 
for any new school adopting it, including allocating dedicated curriculum time, cultural 
changes and whole staff training. 


3. Further refinement of the curriculum and associated resources is required to highlight the 
role, nature and development of exploratory talk and to ensure diverse opportunities for 
oracy, formal and non-formal, are provided. The supporting website requires development 
to provide an effective professional development resource for other schools. 


4. lt was not possible at this stage of development to provide a valid measurement of impact. 
Piloting in other schools and further research would be required prior to a randomised 
control trial of the intervention to establish a stronger evidence base on the impact on oracy 
skills and attainment across subjects. Research is also required on how the intervention is 
interpreted in other schools, and on the opportunities and barriers that arise when 
implementing these approaches at other sites. 


10.1 Limitations 


This evaluation has been conducted alongside the development of the Oracy Curriculum, Culture and 
Assessment Toolkit. The primary purpose of the evaluation was to provide an independent view to 
support the development of the intervention; it is therefore not a full evaluation of the toolkit as it has 
now been developed. Our findings are necessarily partial and indicative and some of the issues we 
raise will have been addressed as part of continuing development work. We are also unable to make 
valid claims about the impact of the intervention at this stage. Our review of the University of 
Cambridge's claims for the validity and reliability of the assessment tool, developed during the project 
lifespan and piloted as a pre and post measure, provides a reasonable degree of confidence in the 
measure. However, during the trial it was only used to measure the progress of 12 students and the 
timing of the pre- and post-tests did not cover the full year. The Raven's Progressive Matrices non- 
verbal reasoning test adopted by the evaluation as an assessment of impact beyond language skills 
has been used in previous research on oracy development (Mercer and Littleton, 2007), but can only 
be considered to be a partial measure of improvements in non-verbal skills. 


10.2 Key conclusions and interpretation 


This developmental project has culminated in the identification of a bounded Oracy Curriculum, 
Culture and Assessment Toolkit that, with some refinement and further development, is ready for 
piloting with other schools. It has potential to make a valuable contribution to the promotion of oracy 
which, research (Mercer, 2008; Alexander, 2001) has suggested, plays a central role in children's 
learning and in students’ current and future lives. 
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Impact 


It was not the intention of this evaluation to assess whether the intervention has a positive impact on 
student attainment. Given the developmental nature of the project it is also too early to make 
confident judgements about the effectiveness of the intervention on students’ oracy skills or learning. 
While there are indicators from the prototype University of Cambridge Oracy Assessment Toolkit, 
used at two points during the year, that the approaches used by School 21 are having a positive 
impact, only 12 School 21 students were tested. The Raven's Progressive Matrices non-verbal 
reasoning test conducted by the evaluation team did not show a significant improvement in students’ 
progress compared to students in a control school. The Raven's Matrices test is only a proxy measure 
of oracy and has strong limitations. It is possible, however, that in part the Raven's Matrices test result 
may also reflect a theme identified during the evaluation, namely a greater focus on engaging in and 
reflecting on talk for presentational purposes than on exploratory talk. Alternatively the result may 
have occurred because the impact of the focus on talk has yet to be realised in students’ ‘talk for 
learning’. However, this does suggest that exploratory talk should be a focus for further scrutiny at 
School 21. 


Below we draw conclusions relating to the key components of the Oracy Curriculum, Culture and 
Assessment Toolkit it is important to note that the components are highly interrelated and when used 
in combination they are mutually reinforcing in terms of the development of oracy. 


Oracy Skills Framework 


Our qualitative research findings indicate that the Oracy Skills Framework has provided an 
appropriate and effective structure to support curriculum design and to underpin the development of 
the Oracy Assessment Toolkit. It also provides the basis for a shared language for teachers 
implementing the Oracy Curriculum, Culture and Assessment Toolkit. However, further research is 
required to see how effective the Oracy Skills Framework is in other contexts and this further review 
may generate the need for further refinement. 


Dedicated Year 7 oracy curriculum 


Given the developmental nature of the project we are unable to arrive at a definitive judgement about 
the efficacy of the dedicated Year 7 oracy curriculum. However, the four units that comprise the 
proposed curriculum are well designed to explore the four strands of oracy skills in the Oracy Skills 
Framework. The distinctive quality of this curriculum is the identification of a set of skills which may be 
used across different contexts, with an emphasis on making appropriate choices about how to draw 
on these appropriately within these different contexts. The explicit oracy focus, and focus on 
reflection, allows for each topic to be examined in depth and each unit culminates in a motivating 
opportunity for the application of skills learned. Particular strengths include: provision for developing 
talk for presentational purposes; the role of drama; and the effective communication of an argument 
or point of view. 


A number of elements remain under development in the curriculum materials and a strengthening of 
the cognitive strand, including provision for exploratory talk, is required. The curriculum materials also 
need to emphasise the need to address oracy within a diverse range of informal as well as formal 
contexts and explore the appropriateness of talk to context. In complementing the dedicated Year 7 
oracy curriculum, which perhaps inevitably focuses on selected types of talk, there is a need to clarify 
how contexts might be provided across the curriculum to ensure students develop broad repertoires 
of talk. There is also a need to recognise that the School 21 curriculum is underpinned by particular 
aims, objectives and principles, and these need to be stated clearly. 
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Oracy in every lesson 


The commitment to promote oracy in every lesson is a strength of the School 21 approach as is the 
emphasis on providing meaningful contexts for talk across the curriculum. Staff and pupils possess 
and utilise a shared language for oracy and are familiar with a range of approaches for organising, 
promoting and reviewing talk. School 21 attribute positive pupil attainment to this strong oracy focus 
and students appear to expect to be involved in group discussion and draw on strategies developed 
during oracy lessons to help them do so. Given what we perceive to be an emphasis on talk for 
presentational purposes in the website materials, it would seem that the development of exploratory 
talk is worth further scrutiny. 


Oracy culture 


The school plans a variety of opportunities for oracy which help to generate a ‘culture of talk’, e.g. 


assemblies, staff development sessions and parents’ evenings, to ensure that talk plays a central 
role. Oracy therefore has high status within the school and there is a shared commitment to oracy 
development by the teaching staff. The development of oracy is closely related to other aspects of the 
school’s curriculum and pedagogy. Other emphases on wellbeing, leadership and enquiry, for 
example, appear to both reinforce and benefit from the oracy curriculum, and the school benefits from 
specialist drama teaching. School 21’s position as a new school with a small student population may 
also make positive engagement by staff and students more likely. This raises questions about the 
extent to which it is possible to translate the School 21 oracy approach to other settings. 


Oracy Assessment Toolkit 


The University of Cambridge Oracy Assessment Toolkit was designed to provide a range of tasks for 
students to demonstrate their oracy skills and ratings scales for teachers and independent reviewers 
to assess oracy attainment. The toolkit provides a variety of contexts for the assessment of different 
kinds of talk and has the potential to offer reliable and valid assessments of students’ oracy skills. 
However, further refinement is needed to increase the reliability of the toolkit. The analyses presented 
by the University of Cambridge team suggest that with appropriately trained raters the assessment 
tool has reasonable reliability. We note though that they have not presented the reliability coefficients 
for all of the assessment tasks. Our own reliability analysis suggests that using the tool with untrained 
raters results in low levels of reliability. The 7-point scale developed during this pilot appears to 
provide more reliable ratings than the original 3-point scale. Further piloting of the Oracy Assessment 
Toolkit with the package of support materials that have been produced in this pilot would enable the 
appropriateness of the 7-point scale to be explored together with further testing of inter-rater reliability. 


It is important to note that the Oracy Assessment Toolkit has been designed as a tool for teachers to 
use to diagnose and track students’ progress and in this respect the evaluation indicates that teachers 
perceive it to be a usable tool. The toolkit would require significant further development and trialling if 
it was intended to be developed to the stage required to provide a standardised test. The exemplar 
rating videos and accompanying commentaries on the University of Cambridge website have strong 
potential for supporting teachers' professional development and stimulating teacher talk and debate 
about oracy. The evaluation highlighted the need for teachers to receive training prior to using the 
toolkit and for some reorganisation of the Oracy Assessment Toolkit website. 


Supporting CPD and resources 


The proposed CPD package to support other schools in the implementation of the Oracy Curriculum, 
Culture and Assessment Toolkit has yet to be tested. Significant progress has been made in setting 
up the Voice 21 website, which provides access to key approaches, teaching resources and 
associated video clips. Further work is now needed to enable the website to underpin CPD for 
schools wishing to develop oracy. This includes structuring the website to foreground the Oracy Skills 
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Framework and providing a commentary that explains the rationale for all materials and how they link 
together. 


10.3 Further development of the intervention and future research 


A key task during this project was for School 21 to identify from its embedded practice a bounded 
intervention that could be shared with other schools. Placing boundaries on the intervention took time 
and it is only towards the end of the project that clarity has emerged around defining what has now 
been termed the Oracy Curriculum, Culture and Assessment Toolkit. We therefore recommend that a 
further pilot is undertaken with a small number of schools. This would enable a more robust 
measurement of impact as well as exploration of how the pilot schools understand and deploy the 
Oracy Curriculum, Culture and Assessment Framework and relate it to existing oracy practices. As we 
have highlighted, some schools may find it difficult to implement the full toolkit in the way that it has 
been embedded in School 21. Piloting would also offer the opportunity to assess the effectiveness of 
the proposed CPD package and the Voice 21 website, and further refine and test the Oracy 
Assessment Toolkit to establish reliability and validity. 


10.4 Recommendations for development of the Oracy, Culture and Curriculum 
Toolkit 


Below we summarise our recommendations for the further development of the Oracy Curriculum, 
Culture and Assessment Toolkit: 


Oracy Skills Framework 


The Oracy Skills framework should continue to be subject to review to assess its appropriateness in 
other settings. 


Guidance for other schools needs to emphasise that: 


e the four strands of the Oracy Skills Framework are closely related and to be effective teaching 
needs to be contextualised and should not overly focus on individual skills in isolation from 
other skills 

e the framework should be used flexibly to support appropriate choices about the direction and 
emphasis of oracy provision in different school contexts. 


Dedicated Year 7 oracy curriculum 


School 21 continues to refer to expertise and approaches developed outside the school to ensure that 
the curriculum takes account of recent research studies and development projects. This would include 
revisiting support for promoting talk for learning, particularly in relation to exploratory talk. 


The dedicated Year 7 curriculum documentation and associated resources be further refined in terms 
of content and presentation. In particular these should: 


e clearly state the aims, objectives and underpinning principles of the dedicated Year 7 
curriculum so that other schools are clear about what the curriculum is, and is not, designed 
to achieve 

e highlight the role, nature and development of exploratory talk 

e emphasise the need to provide a range of motivating and relevant contexts for talk. 


Oracy in every lesson 


School 21 review the extent to which students engage in exploratory talk across subjects and how this 
is promoted across the school, and ensure that the oracy in every lesson documentation and 
resources give greater emphasis to exploratory talk. 
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Oracy culture 


Any scaling-up of the intervention would need to consider how School 21's oracy approach is 
interpreted in other schools, and on the opportunities and barriers that arise when implementing these 
approaches in other sites. 


Oracy Assessment Toolkit 


Further reliability analyses are undertaken particularly on the use of the 7-point ratings scheme as 
well as a proper item-analysis. In order to do this a larger sample of raters would be required. 


The University of Cambridge continue to review the assessment tasks to ensure validity as they are 
used in different contexts. 


Exemplar videos, to support teachers in undertaking reliable assessment and as a basis for teacher 
professional development, are produced. Videos exemplifying achievement at Gold level for the 
Talking Points task would be particularly useful. 


The way in which the Oracy Assessment Toolkit is presented on the University of Cambridge Oracy 
Assessment Toolkit website is reviewed ensuring that: 


e some context to, and rationale for, the Oracy Skills Framework is provided 

e clearer and more detailed descriptions of the contents of each of the task-specific pages are 
presented along with more meaningful named links to documents 

e clear advice on using the assessment sheets to enhance reliability is provided. 


The way in which the Oracy Assessment Toolkit is intended to be used within the overall Oracy 
Curriculum, Culture and Assessment Toolkit intervention is clearly articulated and a clearer 
explanatory link is made between the Voice 21 website and the University of Cambridge website. 


CPD and supporting resources 


The effectiveness of the proposed CPD package should be evaluated when it is rolled out to other 
schools. 


The effectiveness of the Voice21 website could be enhanced by: 


e astructure that foregrounds the Oracy Skills Framework and the other key components of the 
Oracy Curriculum, Culture and Assessment Toolkit and explains the relationship between 
these elements 

e clear rationales and explanations for all materials explaining their purpose and intended use 
in relation to other materials 

e identifying the unique contribution of the School 21 approach as well as acknowledging work 
that is already in place elsewhere and signposting through the site for teachers with differing 
levels of prior knowledge and experience of oracy teaching 

e the addition of materials that exemplify talk for learning. 


As the project progresses it would be helpful to add a section that exemplifies how other schools have 
used the Oracy Curriculum, Culture and Assessment Toolkit. 
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Appendix 1: School 21 and the University of Cambridge 
Oracy Skills Framework 


¢ 1 a) fluency and pace of speech; b) tonal 


PHYSICAL variation; c) clarity of pronunciation; d) voice 
projection 
1. Voice e 2 a) gesture and posture; b) facial expression 


cl ROE Elda and eye contact 


N i 


BINCCLOINT EG ¢ 3 appropriate vocabulary choice 


¢ 4 a) register; b) grammar 


3. Vocabulary e 5 structure and organisation of talk 

Art iet : : 

5 ern laa ¢ 6 rhetorical techniques, such as metaphor, 
6. Rhetorical techniques humour, irony and mimicry 


“ “4 

e 7 a) choice of content to convey meaning and 

ele N ITIVE intention; b) building on the views of others | 

¢ 8 a) seeking information and clarification through 

7. Content questions; b) summarising 

8. Clarifying and ¢ 9 a) maintaining focus on task; b) time management 

Asean e 10 a) giving reasons to support views; b) critically 
examining ideas and views expressed 


10. Reasoning 
11. Audience awareness e 11 taking account of level of understanding of the } 


audience 7 


SOCIAL & 
aVLONNO)\TVE e 12 a) guiding or managing the interactions; b) 


turn-taking 
e 13 listening actively and responding 


12. Working with others ; 
13. Listening and appropriately 


responding e 14 a) self-assurance; b) liveliness and flair 
14. Confidence in 


me opeaking 
eee : oY 


Education Endowment Foundation 62 


Oracy Curriculum, Culture and Assessment Toolkit 


Appendix 2: School 21's mapping of the dedicated oracy 
curriculum to the Oracy Skills Framework 


Unit One: Finding | Unit Two: Unit Three: Unit Four: Ignite 
Our Voice Performance Persuasive Talks 
Poetry Techniques 

Physical Voice appropriate Body language and | Voice appropriate Use of voice to 
for small groups. voice for to formal contexts. | engage listener. 
Proof of listening in | performance. Body language for 
groups. presentation. 

Linguistic Language of Creative and Formal and Passionate 
justification. emotive language. | convincing language. 
Dialogic language. language. Language of 

Questioning justification. 
techniques. Powerful 
Language of arguments. 
justification. Adapt | Persuasion. 
register for Critical language. 
different contexts. 

Cognitive Talk protocols. Scripted, Rapidity of Constructing a 
Pre-talk. Types of | rehearsed response and spoken argument. 
questions. Higher | language. thought process. Time management. 
order thinking. De | Redrafting work. Production of a Redrafting. Peer 
Bono's Hats. spontaneous, critique. 

cognitive argument | Reflection. 
in reaction to what 

is said. 

Summarising the 

debate. 

Social and Groupings. Collaborative Managing Peer critiquing. 

Emotional Listening group work. interactions ina Engagement with 
strategies. Talk Audience formal setting. personal topic. 
protocols. engagement. Listening to others | Engagement with 

in a debate. audience through 
liveliness and flair. 
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Appendix 3: Parental information sheet (intervention) 


Oracy research Project: Parental information sheet 


School 21 and Cambridge University are piloting a programme of activities within the school 
curriculum to support the development of year 7 students’ oracy skills. Sheffield Hallam University has 
been commissioned by the Education Endowment Fund (EEF), a charity dedicated to raising 
educational attainment, to evaluate this project. As part of the evaluation we would like to collect data 
from year 7 students. 


What will the research involve? 


Verbal reasoning tests: All year 7 students will be asked to take a verbal reasoning test in September 
2013 and again in July 2014. This will take about 45 minutes and will take place during the school 
day. The results of the tests will be used to assess the progress made over the year and we will 
compare this to students in another school with similar characteristics. After the evaluation has been 
completed we may share the outcomes of the results of your child's test with School 21 to help them 
support your child. We will only present collated data in our evaluation report and your child's results 
will not be identifiable. 


Observations of teaching and learning, videos of students’ activity and student focus 
groups: Members of the evaluation team will be observing lessons on four days during the year. We 


will also analyse videos recorded by School 21 and Cambridge University of students taking part in 
oracy activities and taking part in a prototype assessment of oracy. We will ask a small number of 
students to take part in a focus group. In the focus group they will be asked about their experiences of 
oracy activities at School 21 and the impact of these activities on their learning. We will not share any 
individual student comments in focus groups with School 21 and no individual student will be 
identifiable in our reporting. 


When will the report be published? 


We will produce a summary project evaluation report that will be available on the Education 
Endowment Fund website (http://educationendowmentfoundation.org.uk/). This report will name 
School 21 but all students will be fully anonymised. We may also use the data for academic and 
professional publications. Again students would be fully anonymised. If you have any questions 
please contact us using the details below. 


What do | do if | don't want my child to take part in the evaluation? 

Participation is voluntary and you can decide to not give your permission to allow your child to 
participate by not returning the consent form attached. Students themselves will also have the 
opportunity to opt out of the verbal reasoning test and all related aspects of the evaluation. 
For any further enquiries please contact: 

Ben Willis (Project Manager): b.willis@shu.ac.uk or tel: 0114 225 6059 


Address: Centre for Education and Inclusion Research, Sheffield Hallam University, Unit 7 Science 
Park, Howard Street, Sheffield S1 1WB 


Sheffield | Centre for 
Hallam ._—| Education and 
University | Inclusion Research 
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Appendix 4: Opt-in parental consent form (intervention) 


Oracy research evaluation: Parental consent form 
Dear parent/carer, 


School 21 and Cambridge University are piloting a programme of activities within the school 
curriculum to support the development of year 7 students’ oracy skills. We (Sheffield Hallam 
University) have been commissioned to evaluate this project. 


As part of the evaluation 


a sample of year 7 students will be asked to take a verbal reasoning test in September 2013 and 
again in July 2014 (the test will take about 45 minutes and take place during the school day) 


we will observe lessons on four days during the year 


we will analyse videos, recorded by School 21 and Cambridge University, of students taking part 
in oracy activities and taking part in a prototype assessment of oracy 


We will ask a small number of students to take part in a focus group about these activities. 


We will not share any individual student comments in focus groups with School 21 and no individual 
student will be identifiable in our reporting. Rest assured no individual child will be identified - all responses 
will be completely confidential and all data stored securely to ensure compliance with the 1998 Data Protection 
Act. No information about individual students will be reported or made available to anyone. After the 
evaluation has been completed we may also share the results of your child's test with School 21 to 
help them support your child. Anonymised data (without names) will be shared with the EEF. 


Participation in the evaluation is voluntary and if at any stage during the project you no longer wish 
your child to take part please contact the evaluation project manager (details below) who will ensure 
no data from your child is used. Students themselves can also choose on the day whether or not to 
take part in the test, focus groups, any observations and request that their data not be used. 


If you ARE willing for your child to take part in the research please complete the slip below and give it 
to your child's teacher or take it to the school office. 


Please return this slip to your child's teacher (or school office). 

Oracy research, | do give my permission for my child to take part in the evaluation. 

Child's full MANNE? 24. secetesesteesecieentdeesdenase cee eeecie sheeseatdencplegeanaddaday end Qedereeansenmertsagsetannsaese 
SIQGMOG snide ti atesncwatseaniestnd aad saiariveeanated wna ciaenaubedee exaceuaeaanednd ena senduenenendunaes oaaedneadias manenacenmenates 
Parent/carer 


PALS ae tio naassatucttad a an caida aantnnd dean nacaneae te damn ctaaaticiad catimeaianmmaneadtmbedaatetial 
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Appendix 5: Information sheet control (School) 


Oracy research Project: information sheet 


Sheffield Hallam University has been commissioned by the Education Endowment Fund (EEF), a 
charity dedicated to raising educational attainment, to evaluate a project focused on piloting a 
programme of activities within the school curriculum to support the development of year 7 students’ 
oracy skills. As part of the evaluation we would like to collect data from year 7 students. 


What would the research involve? 


Verbal reasoning tests: A sample of year 7 students would be asked to take a verbal reasoning test in 
September 2013 and again in July 2014. This would take about 45 minutes and would take place 
during the school day. The results of the tests would be used to assess the progress made over the 
year and would compare this to students in another school with similar characteristics. Participation is 
voluntary and parents would be written to separately and given the opportunity to withdraw their child 
from the evaluation. The students themselves would be given the option to not take part on the day of 
the test. 


When will the report be published? 


We will produce a summary project evaluation report that will be available on the Education 
Endowment Fund website (http://educationendowmentfoundation.org.uk/). We may also use the data 
for academic and professional publications. Again students would be fully anonymised. If you have 
any questions please contact us using the details below. 


What would the benefits be for my school? 


The project should provide evidence about what types of programmes make a difference to students' 
attainment. At the end of the project all schools will be able to access the findings of this evaluation on 
the EEF website. Additionally we would provide an event for staff run by Neil Mercer (an international 
expert on oracy) at Hackney New School to enable the school to benefit from the latest research on 
oracy. 


For any further enquiries please contact: 
Ben Willis (Project Manager): b.willis@shu.ac.uk or tel: 0114 225 6059 


Address: Centre for Education and Inclusion Research, Sheffield Hallam University, 
Unit 7 Science Park, Howard Street, Sheffield S1 1WB 


Sheffield | Centre for 
Hallam, | Education and 
University | Inclusion Research 
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Appendix 6: Information sheet control (parents) 


Information Sheet 


The Education Endowment Fund (EEF) an independent charity dedicated to raising educational 
attainment is supporting a project that is trialling a programme of activities designed to assess and 
improve the oracy skills of year 7 students. The Centre for Education and Inclusion Research at 
Sheffield Hallam University has been commissioned by the EEF to evaluate this project. 


As part of the evaluation the aim is to compare the progress of students in the school that is 
undertaking the trial with student progress in a ‘control’ school with similar characteristics. Hackney 
New School has kindly agreed to act as a control school. 


A sample of year 7 students will be asked to take a verbal reasoning test in September 2013 and 
again in July 2014. The test will take about 45 minutes and will take place during the school day. 


The aim of the project is to provide evidence for all schools about what types of programmes make a 
difference to students’ attainment so they can make informed choices about how best to use their 
resources. On completion of the project all schools will be able to access the findings of this 
evaluation via the Education Endowment Fund website www.educationendowmentfoundation.org.uk . 


Following completion of the project we will also provide an event for staff at the school led by an 
international expert in oracy, to enable the school to benefit from the latest research on oracy. 


What will happen to the information you collect about my child? 


The results of the tests in your school will be compared to the results in the trial school and national 
data on results from these tests. Apart from being available on the website above, the University may 
publish the findings in an academic or professional journal. The name of your school will be 
anonymised and your child's results will not be identifiable. 


All information about children, including test results, will be held confidentially and in compliance with 
the Data Protection Act. After the evaluation has been completed we may also share the results of 

your child's test with your school to help them support your child. Anonymised data (without names) 
will be shared with the Educational Endowment Foundation for current and future research purposes. 


What do | do if | don't want my child to take part in the evaluation? 


Participation is voluntary and you can decide to not give your permission to allow your child to 
participate by returning the consent form attached. Students themselves will have the opportunity to 
opt out of both the verbal reasoning test as well. 


Further information 


Further information is available from Ben Willis who is the Evaluation project manager. Ben can be 
contracted via e-mail b.willis@shu.ac.uk or telephone 0114 25 6059. Our address is, Centre for 
Education and Inclusion Research, Unit 7 Science Park, Sheffield Hallam University, Howard Street, 
S1 1WB 
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Appendix 7: Opt-out control school consent (parent) 


Oracy research: Consent form 
Dear parent/carer, 


The Education Endowment Fund (EEF), a charity dedicated to raising educational attainment, is 
supporting a project that is trialling a programme of activities designed to assess and improve the 
oracy skills of year 7 students. 


Sheffield Hallam University has been commissioned to evaluate this project and we would like to 
compare the progress of students in the school undertaking the trial with a ‘control’ school. Hackney 
New School has kindly agreed to act as a control school. A sample of year 7 students will be asked to 
take a verbal reasoning test in September 2013 and again in June 2014. The test will take about 45 
minutes and will take place during the school day. 


Rest assured no individual child will be identified - all responses will be completely confidential and all 
data stored securely to ensure compliance with the 1998 Data Protection Act. No information about 
individual students will be reported or made available to anyone. After the evaluation has been 
completed we may also share the results of your child's test with Hackney New School to help them 
support your child. Anonymised data (without names) will be shared with the EEF. 


The project should provide evidence about what types of programmes make a difference to students’ 
attainment. At the end of the project all schools will be able to access the findings of this evaluation on 
the EEF website, and we will also be providing an event for staff at Hackney New School to enable 
the school to benefit from the latest research on oracy. Please find attached a project information 
sheet with further information with my contact details should you want to contact me about this 
project. 


If you would prefer your child NOT to take part in the verbal reasoning test, please complete the slip 
below and give it to your child's teacher/ take it to the school office. 


Please return this slip to your child's teacher (or school office). 
Oracy research evaluation 
| do not give my permission for my child to take part in the verbal reasoning test 


Child's full name: 


SIQHE Os 22, sancescrncsmnancauataaae suse ant arunaitiaatnGeacapagiteasaensiaeensacdenseaa instante aeac aaatuatcoteetannin: 
Parent/carer 


DAG ei cancaatan teehee sme tes Santhen sdaee nanan waa anita dies diate aaaiaeaante bran dae au wat eantel aed vee bua ean eaeeauanre 
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Appendix 8: Control school consent (Senior leader) 


Oracy research: Consent form 
Dear Senior leader, 


The Education Endowment Fund (EEF) is supporting a project that is trialling a programme of 
activities designed to assess and improve the oracy skills of year 7 students at an intervention school. 


Sheffield Hallam University has been commissioned to evaluate this project and we would like to 
compare the progress of students in the intervention school with your school which would act as a 
‘control’. A sample of year 7 students would be asked to take a verbal reasoning test in September 
2013 and again in July 2014. The test would take about 45 minutes and take place during the school 
day. 


Rest assured no individual would be identified - all responses would be completely confidential and all 
data stored securely to ensure compliance with the 1998 Data Protection Act. No information about 
individual students would be reported or made available to anyone. After the evaluation we would 
share the results of your student's test with you. Anonymised data would be shared with the EEF. 
Participation is voluntary and parents would be written to separately and given the opportunity to 
withdraw their child from the evaluation. The students themselves would be given the option to not 
take part on the day of the test. 


The project should provide evidence about what types of programmes make a difference to students’ 
attainment. At the end of the project all schools will be able to access the findings of this evaluation on 
the EEF website. Additionally we would provide an event for staff run by Neil Mercer (an international 
expert on oracy) at CONTROL school to enable the school to benefit from the latest research on 
oracy. Please find attached a project information sheet with further information with my contact details 
should you want to contact me about any aspect of this project. 


| give my permission for CONTROL School to be involved in the oracy evaluation and for a sample of 
year 7 students to be tested. 
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Appendix 9: University of Cambridge initial and end 
assessments rating schemes 


A9.1 Rating scale 
For each skill, a student is assessed on a three-way, GOLD/SILVER/BRONZE scale: 


e GOLD means ‘consistently demonstrates this skill’. 
e SILVER means ‘demonstrates this skill some of the time’. 
e BRONZE means ‘rarely or never demonstrates this skill yet’. 
(Mercer et al., 2014: 64) 


A9.1 Lego task rating proforma 


Oracy Skill Student A Student B 


Linguistic 


3 appropriate vocabulary choice 


4 a) register 


Cognitive 


7 a) choice of content to convey meaning and intention 


8 a) seeking information and clarification through 
questions 


9 a) maintaining focus on task 


11 taking account of level of understanding of the 
audience 


Social & Emotional 


12 a) guiding or managing the interactions 


12 b) turn-taking 


13 listening actively and responding appropriately 


Overall assessment 


(Mercer et al., 2014: 64) 
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A9.2 Talking points task rating proforma 


Oracy Skill Student A Student B Student C 


Physical 


1c) clarity of pronunciation 
2 b) facial expression and eye contact 


Linguistic 


4 a) register 


Cognitive 


7 b) building on the views of others 


8 b) summarising 


9 a) maintaining focus on task 


10 a) giving reasons to support views 


10 b) critically examining ideas and views 
expressed 


Social & Emotional 


12 a) guiding or managing the interactions 


12 b) turn-taking 


13 listening actively and responding 
appropriately 


Overall assessment 


(Mercer, Warwick and Ahmed, 2014 p74) 


A9.3 Presentation task rating proforma 


Oracy Skill 
Physical 


1 a) fluency and pace of speech 


1 b) tonal variation 


1c) clarity of pronunciation 


1 d) voice projection 


2 a) gesture and posture 


2 b) facial expression and eye contact 
Linguistic 


3 appropriate vocabulary choice 


4 b) grammar 


5 structure and organisation of talk 


6 rhetorical techniques, such as metaphor, humour, irony and mimicry 


Cognitive 


7 a) choice of content to convey meaning and intention 


9 b) time management 


11 taking account of level of understanding of the audience 
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Social & Emotional 


14 a) self-assurance 


14 b) liveliness and flair 


(Mercer et al., 2014: 80) 
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Appendix 10: University of Cambridge assessment task 
grading conversion to scores 


Cla ale) Numerical 
score 
Bronze 1 
Bronze + 2 
Silver - 3 
Silver 4 
Silver + 5 
Gold - 6 
Gold 7 
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